Joao Pedro Ferreira Canutto
11/25/2025, 7:02 PMkapa.ai
11/25/2025, 7:02 PMJoao Pedro Ferreira Canutto
11/25/2025, 7:03 PM2025-11-25 10:55:56 replication-orchestrator INFO Records read: 375000 (328 MB)
2025-11-25 10:56:16 replication-orchestrator INFO Records read: 380000 (333 MB)
2025-11-25 10:56:37 replication-orchestrator INFO Records read: 385000 (338 MB)
2025-11-25 10:56:41 replication-orchestrator INFO Stream status TRACE received of status: RUNNING for stream tickets
2025-11-25 10:56:41 replication-orchestrator INFO Stream status TRACE with status RUNNING received of sub-type: RATE_LIMITED for stream tickets
2025-11-25 10:56:41 replication-orchestrator INFO Sending update for tickets - RUNNING -> RATE_LIMITED
2025-11-25 10:56:41 replication-orchestrator INFO Stream Status Update Received: tickets - RATE_LIMITED
2025-11-25 10:56:41 replication-orchestrator INFO Updating status: tickets - RATE_LIMITED
2025-11-25 10:56:41 source INFO Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: HTTP Status Code: 429. Error: Too many requests.)
2025-11-25 10:56:41 source INFO Retrying. Sleeping for 35.0 seconds
2025-11-25 10:56:55 replication-orchestrator INFO Records read: 390000 (340 MB)
2025-11-25 10:57:11 replication-orchestrator INFO Records read: 395000 (340 MB)
2025-11-25 10:57:29 replication-orchestrator INFO Records read: 400000 (341 MB)
2025-11-25 10:57:32 replication-orchestrator INFO Sending update for tickets - RATE_LIMITED -> RUNNING
2025-11-25 10:57:32 replication-orchestrator INFO Stream Status Update Received: tickets - RUNNING
2025-11-25 10:57:32 replication-orchestrator INFO Updating status: tickets - RUNNING
2025-11-25 10:57:50 replication-orchestrator INFO Records read: 405000 (346 MB)
2025-11-25 10:58:11 replication-orchestrator INFO Records read: 410000 (351 MB)
2025-11-25 10:58:15 destination ERROR Killed
2025-11-25 10:58:15 replication-orchestrator ERROR DestinationReader error:
2025-11-25 10:58:15 replication-orchestrator INFO DestinationReader finished.
2025-11-25 10:58:15 replication-orchestrator ERROR DestinationWriter error:
2025-11-25 10:58:15 replication-orchestrator ERROR DestinationWriter error:
2025-11-25 10:58:15 replication-orchestrator INFO MessageProcessor finished.
2025-11-25 10:58:15 replication-orchestrator WARN Attempted to close a destination which is already closed.
2025-11-25 10:58:15 replication-orchestrator INFO DestinationWriter finished.
2025-11-25 10:58:15 replication-orchestrator ERROR SourceReader error:
2025-11-25 10:58:15 replication-orchestrator INFO SourceReader finished.
2025-11-25 10:58:15 replication-orchestrator ERROR runJobs failed; recording failure but continuing to finish.
2025-11-25 10:58:15 replication-orchestrator INFO Closing StateCheckSumCountEventHandlerkapa.ai
11/25/2025, 7:03 PMkapa.ai
11/25/2025, 7:03 PMtickets changing to RATE_LIMITED after receiving HTTP status code 429 ("Too many requests"). Airbyte recognizes this and automatically backs off, as indicated by the message: source INFO Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: HTTP Status Code: 429. Error: Too many requests.) source INFO Retrying. Sleeping for 35.0 seconds - This is expected behavior: Airbyte detects rate limiting, pauses, and retries after a delay. This aligns with Airbyte’s recommended error handling for rate limits, which includes using backoff strategies and retrying on 429 errors. The connector will automatically pause and resume once the rate limit is lifted, as seen when the status returns to RUNNING for the tickets stream [Common Error Handling Patterns](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/error-handling#common-error-handling-patterns), [Connection Troubleshooting](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/review-connection-status#connection-troubleshooting). 2. Sync Progress - Despite the rate limiting, the sync continues to read records, indicating that Airbyte is able to recover from the rate limit event and proceed with data extraction. 3. Destination Errors - Later in the logs, there are several errors related to the destination: destination ERROR Killed replication-orchestrator ERROR DestinationReader error: replication-orchestrator ERROR DestinationWriter error: - The logs do not provide details about the cause of these destination errors. The knowledge sources do not contain enough information to definitively diagnose these errors from the provided log snippet. ### Summary - Rate limiting is being handled as expected by Airbyte, with automatic backoff and retry on 429 errors. - The sync resumes after the rate limit is lifted, as indicated by the status updates and continued record reading. - Destination errors occur later, but the logs do not specify the root cause, and the knowledge sources do not provide enough detail to analyze these errors further. If you need to troubleshoot the destination errors, you may need to review more detailed destination logs or configuration, as the current knowledge sources do not explicitly address this scenario. References integrated above.Joao Pedro Ferreira Canutto
11/25/2025, 7:07 PM2025-11-25 10:30:53 platform INFO ----- END DISCOVER -----
2025-11-25 10:30:53 platform INFO
2025-11-25 10:30:54 platform INFO APPLY Stage: BUILD — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync)
2025-11-25 10:30:54 platform INFO APPLY Stage: CLAIM — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync)
2025-11-25 10:30:54 platform INFO Claimed: true for workload f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync via API in dataplane AUTOe067ebd9-9266-4cf3-a59d-87c949337c8a (87ecdcd2-c5a4-4233-a6cf-f61c659b87ec)
2025-11-25 10:30:54 platform INFO APPLY Stage: LOAD_SHED — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync)
2025-11-25 10:30:54 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync)
2025-11-25 10:30:54 platform INFO No pod found running for workload f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync
2025-11-25 10:30:54 platform INFO APPLY Stage: MUTEX — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync)
2025-11-25 10:30:54 platform INFO Mutex key: f82abdad-4c0d-438c-b4bd-eb658ddfade0 specified for workload: f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync. Attempting to delete existing pods...
2025-11-25 10:30:54 platform INFO Mutex key: f82abdad-4c0d-438c-b4bd-eb658ddfade0 specified for workload: f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync found no existing pods. Continuing...
2025-11-25 10:30:54 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync)
2025-11-25 10:30:54 platform INFO APPLY Stage: LAUNCH — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync)
2025-11-25 10:30:54 platform INFO [initContainer] image: airbyte/workload-init-container:1.8.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})
2025-11-25 10:30:54 platform INFO Launching replication pod: replication-job-23712-attempt-0 (selectors = {}) with containers:
2025-11-25 10:30:54 platform INFO [source] image: airbyte/source-zendesk-support:4.10.12 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
2025-11-25 10:30:54 platform INFO [destination] image: airbyte/destination-aws-datalake:0.1.58 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
2025-11-25 10:30:54 platform INFO [orchestrator] image: airbyte/container-orchestrator:1.8.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})
2025-11-25 10:31:08 platform INFO Attempting to update workload: f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync to LAUNCHED.
2025-11-25 10:31:08 platform INFO Pipeline completed for workload: f82abdad-4c0d-438c-b4bd-eb658ddfade0_23712_0_sync.kapa.ai
11/25/2025, 7:07 PMkapa.ai
11/25/2025, 7:07 PMairbyte/source-zendesk-support:4.10.12) and destination (airbyte/destination-aws-datalake:0.1.58) containers are being launched with the following resource requirements: - Limits: memory=2Gi, cpu=1 - Requests: memory=1Gi, cpu=0.2 The orchestrator and init containers have similar or slightly higher resource requests and limits. - Pod Launch Sequence: The logs detail the standard Airbyte pod launch sequence, including stages like CLAIM, LOAD_SHED, CHECK_STATUS, MUTEX, ARCHITECTURE, and LAUNCH. No errors or warnings are present in this sequence, and the pod is successfully launched. - Resource Logging: The resource requirements for each container are logged as expected. This matches Airbyte’s documented behavior, where resource requirements are logged as part of the job logs for debugging and confirmation purposes. You can use these logs to verify that the intended resource configuration is being applied to your sync jobs, as described in the documentation [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#debugging). - No Issues Detected: There are no indications of resource allocation problems, pod scheduling failures, or other errors in the provided log snippet. The configuration appears to be applied correctly. Summary: The pod configuration logs confirm that your Airbyte sync job is being launched with the specified resource requests and limits for each container. This matches the expected behavior described in the Airbyte documentation, and there are no errors or misconfigurations evident in the logs you provided. If you are troubleshooting resource issues, you should continue to monitor these logs to ensure the correct configuration is being detected and applied during job execution, as recommended in the documentation. If you have a specific concern or are experiencing issues not visible in these logs, please provide additional context or error messages for further analysis.Joao Pedro Ferreira Canutto
11/25/2025, 7:11 PM2025-11-25 10:58:15 destination ERROR Killed and my concern is that the resources allocated for the pod are insufficient when getting near 400MB:
2025-11-25 10:57:50 replication-orchestrator INFO Records read: 405000 (346 MB)
2025-11-25 10:58:11 replication-orchestrator INFO Records read: 410000 (351 MB)
2025-11-25 10:58:15 destination ERROR Killedkapa.ai
11/25/2025, 7:11 PMkapa.ai
11/25/2025, 7:11 PMdestination ERROR Killed is due to the destination container running out of memory. In Kubernetes environments, when a container exceeds its memory limit (in your case, 2Gi), it is terminated by the system, often resulting in a "Killed" message in the logs. This is consistent with Airbyte's documented behavior: if a job is running out-of-memory, you should check the job logs to confirm the resource configuration and look for OOM (Out Of Memory) events or "Killed" errors in the destination or source logs [Configuring Connector Resources - Debugging](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#debugging). Your pod configuration shows both the source and destination have a memory limit of 2Gi. As the sync approaches 400MB of data processed, the destination container is killed, which strongly suggests it is hitting this memory ceiling. This matches common Airbyte troubleshooting advice: if you see containers being killed, especially during large syncs, you should increase the memory limits for the affected connector(s) [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources). Recommendation: Increase the memory limit for your destination connector in your resource configuration (e.g., in your values.yaml or via connector-specific overrides). Monitor the logs after making this change to confirm the container is no longer being killed due to memory exhaustion. If you continue to see the "Killed" error after increasing memory, further investigation into the connector's memory usage pattern may be needed, but the knowledge sources strongly indicate insufficient memory as the root cause in this scenario.Joao Pedro Ferreira Canutto
11/26/2025, 8:44 PMselect * from actor_definition where actor_definition.docker_repository like '%<image-name>';
But this command generates this result:
db-airbyte=# select * from actor_definition where actor_definition.docker_repository like '%zendesk';
ERROR: column actor_definition.docker_repository does not exist
LINE 1: select * from actor_definition where actor_definition.docker...
I don't know if this matters, but my Self-Managed Community airbyte 1.8.1 installation uses kubernetes.kapa.ai
11/26/2025, 8:44 PMkapa.ai
11/26/2025, 8:44 PMactor_definition table using the docker_repository column, as shown: sql select * from actor_definition where actor_definition.docker_repository like '%<image-name>'; and then update the resource requirements using the id from that result: sql update actor_definition set resource_requirements = '{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "2", "cpu_request": "2", "memory_limit": "2048Mi", "memory_request": "2048Mi"}}]}' where id = '<id-from-step-1>'; [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources) However, you are encountering the error: ERROR: column actor_definition.docker_repository does not exist ### Analysis Based on the Airbyte database schema documentation for version 1.8, the actor_definition table does not list a docker_repository column. The documented columns are: id, name, icon, actor_type, source_type, created_at, updated_at, tombstone, resource_requirements, public, custom, max_seconds_between_messages, default_version_id, icon_url, metrics, and enterprise [Database Data Catalog - actor_definition](https://docs.airbyte.com/platform/1.8/understanding-airbyte/database-data-catalog#actor_definition). The docker_repository column actually exists in the actor_definition_version table, not in actor_definition [Database Data Catalog - actor_definition_version](https://docs.airbyte.com/platform/1.8/understanding-airbyte/database-data-catalog#actor_definition_version). ### What to do To find the connector definition for a specific image, you should query the actor_definition_version table, for example: sql select * from actor_definition_version where docker_repository like '%zendesk%'; Then, use the actor_definition_id from the result to update the resource_requirements in the actor_definition table. ### Summary - The documentation instructs you to query a column (docker_repository) in actor_definition that does not exist according to the schema. - You should query docker_repository in the actor_definition_version table instead, and then use the actor_definition_id to update the actor_definition table. If you need to update the resource requirements, the correct workflow based on the schema is: 1. Query actor_definition_version for the docker_repository. 2. Use the resulting actor_definition_id to update the resource_requirements in actor_definition. The knowledge sources do not explain why the documentation references the wrong column, but this is the correct approach based on the schema for Airbyte 1.8.1. --- If you need further help with the exact SQL commands, let me know!Joao Pedro Ferreira Canutto
11/26/2025, 9:10 PMkapa.ai
11/26/2025, 9:10 PMkapa.ai
11/26/2025, 9:10 PMactor_definition_id for a custom connector created with the Builder is not explicitly detailed. The knowledge sources describe how to query the actor_definition_version table using the docker_repository column for official connectors, but they do not provide specific instructions for locating custom connectors created via the Builder. However, the actor_definition table contains a custom boolean column, which indicates if the connector is user-defined. You can try to identify your custom connector by querying for entries where custom = true and filtering by the connector name (e.g., 'Zendesk Exports'): sql select * from actor_definition where custom = true and name = 'Zendesk Exports'; This should return the row for your custom connector, including its id, which is the actor_definition_id you need to update the resource_requirements. If you do not see your custom connector in the results, the knowledge sources do not provide further details on alternative methods for locating custom Builder connectors in the database. Summary: - Query the actor_definition table for entries where custom = true and name matches your connector. - The id column in the result is your actor_definition_id. The knowledge sources do not provide more specific guidance for custom Builder connectors beyond this approach.Joao Pedro Ferreira Canutto
11/26/2025, 9:35 PM2025-11-26 18:30:43 platform INFO ----- END DISCOVER -----
2025-11-26 18:30:43 platform INFO
2025-11-26 18:30:44 platform INFO APPLY Stage: BUILD — (workloadId=da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync)
2025-11-26 18:30:44 platform INFO APPLY Stage: CLAIM — (workloadId=da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync)
2025-11-26 18:30:44 platform INFO Claimed: true for workload da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync via API in dataplane AUTOe067ebd9-9266-4cf3-a59d-87c949337c8a (87ecdcd2-c5a4-4233-a6cf-f61c659b87ec)
2025-11-26 18:30:44 platform INFO APPLY Stage: LOAD_SHED — (workloadId=da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync)
2025-11-26 18:30:44 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync)
2025-11-26 18:30:44 platform INFO No pod found running for workload da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync
2025-11-26 18:30:44 platform INFO APPLY Stage: MUTEX — (workloadId=da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync)
2025-11-26 18:30:44 platform INFO Mutex key: da4a36f8-8893-42be-8066-f5a1714145c4 specified for workload: da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync. Attempting to delete existing pods...
2025-11-26 18:30:44 platform INFO Mutex key: da4a36f8-8893-42be-8066-f5a1714145c4 specified for workload: da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync found no existing pods. Continuing...
2025-11-26 18:30:44 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync)
2025-11-26 18:30:44 platform INFO APPLY Stage: LAUNCH — (workloadId=da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync)
2025-11-26 18:30:44 platform INFO [initContainer] image: airbyte/workload-init-container:1.8.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})
2025-11-26 18:30:44 platform INFO Launching replication pod: replication-job-24278-attempt-0 (selectors = {}) with containers:
2025-11-26 18:30:44 platform INFO [source] image: airbyte/source-declarative-manifest:6.61.6 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.5}, additionalProperties={})
2025-11-26 18:30:44 platform INFO [destination] image: airbyte/destination-aws-datalake:0.1.58 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.5}, additionalProperties={})
2025-11-26 18:30:44 platform INFO [orchestrator] image: airbyte/container-orchestrator:1.8.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})
2025-11-26 18:30:59 platform INFO Attempting to update workload: da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync to LAUNCHED.
2025-11-26 18:30:59 platform INFO Pipeline completed for workload: da4a36f8-8893-42be-8066-f5a1714145c4_24278_0_sync.
2025-11-26 18:31:11 replication-orchestrator INFO Running replication worker...
2025-11-26 18:31:11 replication-orchestrator INFO Starting replication worker. job id: 24278 attempt: 0
2025-11-26 18:31:11 replication-orchestrator INFO
2025-11-26 18:31:11 replication-orchestrator INFO ----- START REPLICATION -----kapa.ai
11/26/2025, 9:35 PMkapa.ai
11/26/2025, 9:35 PMresource_requirements in the actor_definition table for a connector, no Airbyte service or pod restart is required for the change to take effect. The new resource requirements should be picked up the next time a new sync job is launched, as Airbyte reads the configuration from the database when creating new job pods. This is confirmed by the documentation, which describes updating the database and then running a sync, with no mention of needing to restart services or pods for the changes to apply [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources). However, if you are not seeing the updated resource allocations reflected in the pod logs after making the change, consider the following possibilities based on the knowledge sources: - Precedence of Resource Configuration: If there are more specific resource requirements set at the connection level, these will override the connector definition settings. The configuration hierarchy is: instance-wide < connector definition < connector-specific < connection-specific, with connection-specific having the highest precedence. Make sure there are no overrides at a narrower scope that would take precedence over your connector definition update [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources). - Log Verification: You can confirm which resource requirements are being applied by checking the job logs. Airbyte logs the resource requirements for each container as the pod is launched. Look for lines like: [source] image: ... resources: ResourceRequirements(claims=[], limits={memory=...}, requests={memory=...}, ...) to verify the actual values being used [Debugging Connection Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#debugging). If you have confirmed there are no more specific overrides and the logs still do not reflect your changes, the knowledge sources do not provide further troubleshooting steps for this scenario. Summary: - No restart is required; changes should apply on the next sync. - Check for more specific resource overrides. - Verify applied resources in the job logs. If the issue persists after these checks, the knowledge sources do not provide additional guidance.Joao Pedro Ferreira Canutto
11/26/2025, 10:02 PMdb-airbyte=# select * from actor_definition where id = '79c1aa37-dae3-42ae-b333-d1c105477715';
-[ RECORD 1 ]----------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
id | 79c1aa37-dae3-42ae-b333-d1c105477715
name | Zendesk Support
icon | zendesk-support.svg
actor_type | source
source_type | api
created_at | 2025-08-26 20:28:52.89926+00
updated_at | 2025-10-24 20:41:47.500245+00
tombstone | f
resource_requirements | {"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "2", "cpu_request": "2", "memory_limit": "10Gi", "memory_request": "10Gi"}}]}
public | t
custom | f
max_seconds_between_messages | 60
default_version_id | 159d21da-d1aa-4ec3-b23d-a2c27a66cc95
icon_url | https://connectors.airbyte.com/files/metadata/airbyte/source-zendesk-support/latest/icon.svg
metrics | {"all": {"usage": "high", "connector_name": "Zendesk Support", "connector_type": "source", "airbyte_platform": "all", "connector_version": "all", "docker_repository": "airbyte/source-zendesk-support", "sync_success_rate": "high", "connector_definition_id": "79c1aa37-dae3-42ae-b333-d1c105477715"}, "oss": {"usage": "high", "connector_name": "Zendesk Support", "connector_type": "source", "airbyte_platform": "oss", "connector_version": "4.10.6", "docker_repository": "airbyte/source-zendesk-support", "sync_success_rate": "high", "connector_definition_id": "79c1aa37-dae3-42ae-b333-d1c105477715"}, "cloud": {"usage": "high", "connector_name": "Zendesk Support", "connector_type": "source", "airbyte_platform": "cloud", "connector_version": "4.10.6", "docker_repository": "airbyte/source-zendesk-support", "sync_success_rate": "high", "connector_definition_id": "79c1aa37-dae3-42ae-b333-d1c105477715"}}
enterprise | f
But when running the sync I'm getting:
2025-11-26 18:46:27 platform INFO [source] image: airbyte/source-zendesk-support:4.10.12 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
Before the change, the field resource_requirements was empty. How can I find out if there are any connector or connection specific configurations overwriding this configuration?kapa.ai
11/26/2025, 10:02 PMkapa.ai
11/26/2025, 10:02 PMactor_definition (connector definition) settings, you should check the following, based on the configuration hierarchy described in the documentation: 1. Connection-Specific Resource Requirements These have the highest precedence. You can check for them by querying the connection table for the relevant connection(s) and inspecting the resource_requirements field: sql select id, name, resource_requirements from connection where source_id = '<source-actor-id>'; If resource_requirements is set for a connection, it will override the connector definition settings for that sync [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#configuring-connection-specific-requirements). 2. Connector-Specific Resource Requirements These apply to a specific connector instance (actor) and also override the connector definition. You can check the actor table for your source: sql select id, name, resource_requirements from actor where actor_definition_id = '79c1aa37-dae3-42ae-b333-d1c105477715'; If resource_requirements is set here, it will take precedence over the definition-level configuration. 3. Precedence Order The order of precedence is: - Instance-wide (lowest) - Connector definition - Connector-specific - Connection-specific (highest) So, if either the actor or connection tables have resource_requirements set, those will override your changes in actor_definition [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources). Summary: Check the resource_requirements fields in both the actor and connection tables for your connector and its connections. If you find values set there, those are overriding your connector definition settings. If you do not find any overrides, but the pod is still not picking up your changes, the knowledge sources do not provide further troubleshooting steps for this scenario.Joao Pedro Ferreira Canutto
11/26/2025, 10:21 PMdb-airbyte=# select id, name, resource_requirements from actor where actor_definition_id = '79c1aa37-dae3-42ae-b333-d1c105477715';
-[ RECORD 1 ]---------+-------------------------------------
id | 75a0f6df-41d1-47ec-bd01-6b1f9ae93545
name | Zendesk Support
resource_requirements | {}
db-airbyte=# select id, name, resource_requirements from connection where source_id = '75a0f6df-41d1-47ec-bd01-6b1f9ae93545';
-[ RECORD 1 ]---------+-------------------------------------------------
id | a105600d-3b43-4c67-b7cc-ffde80262e4a
name | Zendesk Support → DEV PARQUET AWS Datalake
resource_requirements | {}
-[ RECORD 2 ]---------+-------------------------------------------------
id | f82abdad-4c0d-438c-b4bd-eb658ddfade0
name | Zendesk Totvs Suporte → aws_datalake_parquet_prd
resource_requirements | null
On the connection table the connection I'm looking at and thats showing still 2Gi instead of the 10Gi I put is the second record, with null for the resource_requirementskapa.ai
11/26/2025, 10:21 PMkapa.ai
11/26/2025, 10:21 PMresource_requirements fields in the actor and connection tables are {} or null, there should be no connector-specific or connection-specific overrides in place. In this case, the resource requirements from the actor_definition table (your connector definition) should apply to your sync jobs [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources). However, your logs still show the old resource allocation (2Gi instead of 10Gi). The knowledge sources do not provide a reason for this discrepancy if there are no overrides present. They also do not mention any required restart or cache invalidation step after updating the database—changes should take effect on the next sync job. Summary: - No overrides are present in your actor or connection tables. - The sync job should use the resource_requirements from the actor_definition table. - If the new limits are not being applied, the knowledge sources do not provide further troubleshooting steps for this scenario. If you continue to experience this issue, it may be a bug or undocumented behavior, as the expected precedence and update process are not resulting in the correct resource allocation according to the official documentation.Joao Pedro Ferreira Canutto
11/26/2025, 10:27 PMkapa.ai
11/26/2025, 10:27 PMkapa.ai
11/26/2025, 10:28 PMresource_requirements annotation you provided is correct in structure: json { "jobSpecific": [ { "jobType": "sync", "resourceRequirements": { "cpu_limit": "2", "cpu_request": "2", "memory_limit": "10Gi", "memory_request": "10Gi" } } ] } Regarding the units for memory, the documentation and community sources show examples using both "2048Mi" and `"2g"`/`"2Gi"` for memory values. For example, the documentation uses "2048Mi" in its SQL update examples, but community answers confirm that "2g" or "2Gi" are also accepted and interpreted correctly by Airbyte and Kubernetes, as these are standard Kubernetes units [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources), [Forum: How to specify CPU,MEMORY limit in .env](https://discuss.airbyte.io/t/1675). Summary: - Your annotation is correct in structure. - Both "Mi" and "Gi" units are supported for memory, so "10Gi" is valid. If you want to be fully consistent with the documentation examples, you can use "Mi" (e.g., "10240Mi" for 10Gi), but "10Gi" is also valid and should work as expected. The knowledge sources do not indicate any requirement to use only "Mi" units.Joao Pedro Ferreira Canutto
11/26/2025, 10:32 PMkapa.ai
11/26/2025, 10:32 PMkapa.ai
11/26/2025, 10:32 PMvalues.yaml file using environment variables such as JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT. These apply to all job containers unless overridden by more specific settings. 2. Connector definition: Set in the actor_definition table’s resource_requirements field. This applies to all connectors of that type unless overridden by connector-specific or connection-specific settings. 3. Connector-specific (actor) configuration: Set in the actor table’s resource_requirements field. This applies to a specific connector instance and overrides both instance-wide and connector definition settings. 4. Connection-specific configuration: Set in the connection table’s resource_requirements field. This has the highest precedence and overrides all other settings. The documentation does not mention any other configuration files, environment variables, or Kubernetes annotations (outside of those set via values.yaml or the Airbyte database) that would override these resource allocations in a Self-Managed Community installation. The configuration hierarchy is clearly defined, and if all of these are empty or unset, the instance-wide defaults apply. [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources) If you are still seeing unexpected resource allocations, the knowledge sources do not provide further details on additional override mechanisms for memory resources in this setup.Joao Pedro Ferreira Canutto
11/26/2025, 10:35 PMkapa.ai
11/26/2025, 10:35 PMkapa.ai
11/26/2025, 10:35 PMvalues.yaml file used for your Helm deployment. Specifically, look for the following environment variables: - JOB_MAIN_CONTAINER_MEMORY_REQUEST - JOB_MAIN_CONTAINER_MEMORY_LIMIT - JOB_MAIN_CONTAINER_CPU_REQUEST - JOB_MAIN_CONTAINER_CPU_LIMIT These variables define the default (instance-wide) resource requests and limits for all job containers unless overridden by more specific settings. You can find and review these settings in your values.yaml file under the global.env_vars section. For example: yaml global: env_vars: JOB_MAIN_CONTAINER_MEMORY_REQUEST: 1Gi JOB_MAIN_CONTAINER_MEMORY_LIMIT: 2Gi JOB_MAIN_CONTAINER_CPU_REQUEST: 0.5 JOB_MAIN_CONTAINER_CPU_LIMIT: 1 If these variables are not set, Airbyte will use its internal defaults. Remember, these instance-wide settings have the lowest precedence and can be overridden by connector definition, connector-specific, or connection-specific resource requirements [Configuring Connector Resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources). If you do not find these variables in your values.yaml, they are likely unset and not influencing your resource allocations.