Ishan Anilbhai Koradiya
11/26/2025, 3:54 PMkapa.ai
11/26/2025, 3:54 PMPending with “Insufficient cpu” or OOMs, and increasing node resources (more vCPUs / RAM) or pod limits/requests resolved the issue. [[Mysql issue](https://github.com/airbytehq/airbyte/issues/68104); [MongoDB failure](https://discuss.airbyte.io/t/476)] - There are also cases where worker/resource problems cause confusing behavior: jobs fail to start pods or hit OOMs but parts of the system still mark workflows as completed or “success” from Temporal’s perspective. [[Resource leak](https://discuss.airbyte.io/t/2309); [Replication failure origin](https://discuss.airbyte.io/t/3619)] Why it can look “successful”: - The Temporal workflow driving the job may complete its control-flow steps (e.g., finishing the workload-launcher pipeline), so the high-level job status may be “succeeded”, even though the replication pod never actually ran or ran only partially. - The log line you pasted is from the platform / workload-launcher layer, not necessarily the final job status computation. What you should check / do next: 1. Look at the replication pod for that job: - kubectl get pods -n <namespace> and find the replication-job-<job>-attempt-<n> pod. - kubectl describe pod ... and check Events for “Insufficient cpu/memory”, OOMKilled, or scheduling failures. Similar symptoms have been reported and fixed by increasing CPU on the node. [[Mysql issue](https://github.com/airbytehq/airbyte/issues/68104)] 2. Confirm actual job outcome in Airbyte UI / API: - Open the job in the Airbyte UI and confirm: - Was any data actually written? - Are there retry attempts or partial-success logs? 3. Check and adjust resource settings: - On Kubernetes / Helm, ensure your workload / replication resource requests and limits are sane, and not conflicting with any per-connector overrides in the DB. There is a known issue where actor-specific resource settings can override your global workloads.resources and cause mismatches like “requested 2Gi while limit is 500Mi”. [[workload resources](https://github.com/airbytehq/airbyte/issues/68162)] - If you see pods failing to start due to insufficient resources, either: - Increase node capacity (more vCPUs / memory), or - Lower the per-job requests/limits, or - Run fewer concurrent syncs. 4. Look for repeated ResourceConstraintException messages: - If this happens regularly, it’s an indication your cluster is under-provisioned for the number/size of syncs you’re running. Because the knowledge sources don’t contain a specific note explaining why such jobs are always marked successful in this exact scenario, I can’t say definitively whether this is a UI/Temporal-status bug or expected edge behavior; but all available evidence points to resource exhaustion preventing the replication pod from starting while the control workflow still completes. If you can share: - The job ID, - The kubectl describe output of the replication pod for that job, - Your Helm workloads.resources (or job resources) settings, then you can compare them directly with the patterns in the issues above to confirm it’s the same resource-constraint situation.Ishan Anilbhai Koradiya
11/26/2025, 4:38 PM'{"cpu_limit": "3", "cpu_request": "2", "memory_limit": "4Gi", "memory_request": "3Gi"}'
Why would it not schedule on a new node if not availbel on current ones ?kapa.ai
11/26/2025, 4:38 PMkapa.ai
11/26/2025, 4:39 PMUnable to start the REPLICATION pod or “Invalid resource requirements detected” when limits/requests for the actor are inconsistent or overridden by existing DB values, regardless of cluster autoscaling. [[workload resources issue](https://github.com/airbytehq/airbyte/issues/68162)] The knowledge sources do not explain Kubernetes autoscaler behavior (e.g., when it chooses to add nodes, pod priority, taints, PDBs, unschedulable pods, etc.), nor how your specific autoscaling policy is configured. They only state that: - You must ensure the K8s cluster can schedule up to 2 x <number-of-possible-concurrent-connections> pods. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes)] - Resource constraints at the cluster level can prevent pods from starting, causing resource-related job failures. [[Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#connector-cpu--memory-settings)] So, based on the docs: - Airbyte is behaving as designed: it tries to launch a REPLICATION pod with your specified cpu_request=2, cpu_limit=3, memory_request=3Gi, memory_limit=4Gi. - If Kubernetes (plus your cluster autoscaler configuration) does not schedule or scale out to satisfy that pod, Airbyte will see it as a resource failure and raise ResourceConstraintException. - The documentation does not say why Kubernetes/your autoscaler would not add a new node for such a pod; that depends on your K8s / autoscaler settings and is outside what the Airbyte docs cover. To move forward, the Airbyte-side actions the docs recommend are: 1. Verify that the cluster in practice can handle the total number of concurrent pods and their requests; otherwise, reduce concurrent syncs or connector resource requests. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes); [Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)] 2. Check for per-connector DB overrides that might be setting different, possibly invalid resource requirements compared to your global settings. [[workload resources issue](https://github.com/airbytehq/airbyte/issues/68162)] Anything beyond that (e.g., “why exactly didn’t the autoscaler create a node?”) is not described in the Airbyte knowledge sources, so I can’t answer that part confidently from this material alone.Ishan Anilbhai Koradiya
11/26/2025, 5:12 PMkapa.ai
11/26/2025, 5:13 PMkapa.ai
11/26/2025, 5:13 PMvalues.yaml, e.g. JOB_MAIN_CONTAINER_CPU_REQUEST, JOB_MAIN_CONTAINER_CPU_LIMIT, JOB_MAIN_CONTAINER_MEMORY_REQUEST, JOB_MAIN_CONTAINER_MEMORY_LIMIT. These apply to all job containers unless overridden. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-instance-wide-requirements)] - Connector definition: all sources/destinations of a given type (e.g. all MongoDB sources) via the actor_definition.resource_requirements field in the DB. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-definition-requirements)] - Connector‑specific: one specific connector instance (e.g. one MongoDB source) — available via UI/API for Self‑Managed Enterprise only. [[Configuring connector‑specific requirements](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-specific-requirements)] - Connection‑specific: one specific source–destination pair (the connection). Precedence (highest wins): Connection > Connector > Connector definition > Instance‑wide. [[1.5 release notes](https://docs.airbyte.com/release_notes/v-1.5#-self-managed-enterprise-changes)] 2. Per‑connector (source vs destination) Because connector‑specific and connector‑definition configs are tied to a specific actor (source or destination), you can give: - one resource profile to the source connector, and - a different profile to the destination connector, for the same connection. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources)] In Enterprise, you do this via the “Connector resource allocation” dropdown on each connector’s Settings tab or via API. [[Configuring connector‑specific requirements](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-specific-requirements)] 3. “Main sync job” / orchestrator vs read/write containers Instance‑wide env vars (JOB_MAIN_CONTAINER_*) apply to job containers, including the orchestrator pods. [[Orchestrator limits](https://discuss.airbyte.io/t/3775)] The docs and forum answer explicitly say the orchestrator gets its resource requirements from JOB_MAIN_CONTAINER_* and there is no separate documented set of env vars to configure orchestrator vs read/write differently. [[Orchestrator limits](https://discuss.airbyte.io/t/3775)] So, in practice, with what’s documented: - You can: - Set different resources for source vs destination (per connector or connector definition, Enterprise gives a UI for this). - Override per connection when needed. - Control global defaults for all job containers with JOB_MAIN_CONTAINER_*. - You cannot (per docs): - Independently tune “main sync job/orchestrator” vs “read” vs “write” containers with three distinct resource knobs; orchestrator uses the same JOB_MAIN_CONTAINER_* settings. [[Orchestrator limits](https://discuss.airbyte.io/t/3775)] If you’re on Self‑Managed Community/Core, connector‑specific UI controls are not available; you’re limited to instance‑wide env vars and DB‑level `actor_definition.resource_requirements`/connection overrides. [[Connector‑specific requirements](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-specific-requirements)]Ishan Anilbhai Koradiya
11/26/2025, 5:14 PMkapa.ai
11/26/2025, 5:14 PMkapa.ai
11/26/2025, 5:14 PMJOB_MAIN_CONTAINER_* env vars, set under global.env_vars in values.yaml (Helm v1) or equivalent. These apply to all job containers (orchestrator + read/write) unless overridden by DB‑level settings. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-instance-wide-requirements)] Example `values.yaml`: yaml global: env_vars: JOB_MAIN_CONTAINER_CPU_REQUEST: "2" JOB_MAIN_CONTAINER_CPU_LIMIT: "3" JOB_MAIN_CONTAINER_MEMORY_REQUEST: "3Gi" JOB_MAIN_CONTAINER_MEMORY_LIMIT: "4Gi" This is the documented way to configure instance‑wide sync job resources when deploying via Helm. [[OOM issue example](https://github.com/airbytehq/airbyte/issues/48816)] To set different resources for source vs destination vs a particular connection, you cannot do that purely in Helm; you must use the DB‑backed resource requirement fields (actor_definition.resource_requirements, connector‑specific, or connection‑specific overrides) as described in the connector resources guide. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-definition-requirements)] The docs do not provide Helm values for separately tuning orchestrator/read/write or per‑connector resources.Ishan Anilbhai Koradiya
11/26/2025, 5:18 PM2 x <number-of-possible-concurrent-connections> pods at once. This is the worse case estimate, and most users should be fine with 2 x <number-of-possible-concurrent-connections> as a rule of thumb.kapa.ai
11/26/2025, 5:18 PMkapa.ai
11/26/2025, 5:18 PM2 × concurrent_syncs helps avoid unschedulable pods and resource errors. [[What to scale](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)]Ishan Anilbhai Koradiya
11/26/2025, 5:21 PMkapa.ai
11/26/2025, 5:21 PMkapa.ai
11/26/2025, 5:21 PM