Hi <@U04S8PLJWMT> in one of my mongo db sync, i se...
# ask-ai
i
Hi @kapa.ai in one of my mongo db sync, i see this error but the pipeline is marked successful. Why would this be happening ? 2025-11-26 020051 platform ERROR Stage Pipeline Exception: io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.ResourceConstraintException: Unable to start the REPLICATION pod. This may be due to insufficient system resources. Please check available resources and try again. message: io.airbyte.workers.exception.ResourceConstraintException: Unable to start the REPLICATION pod. This may be due to insufficient system resources. Please check available resources and try again. stackTrace: [Ljava.lang.StackTraceElement;@5ea79bf4
k
This error almost certainly indicates a resource issue at the Kubernetes / host level, even though the sync is being marked as successful. A few relevant points from similar reports: - When replication pods cannot be scheduled or started due to lack of CPU/memory, Airbyte surfaces errors like “Unable to start the REPLICATION pod. This may be due to insufficient system resources.” [[Mysql issue](https://github.com/airbytehq/airbyte/issues/68104)] - Users have seen replication pods stuck
Pending
with “Insufficient cpu” or OOMs, and increasing node resources (more vCPUs / RAM) or pod limits/requests resolved the issue. [[Mysql issue](https://github.com/airbytehq/airbyte/issues/68104); [MongoDB failure](https://discuss.airbyte.io/t/476)] - There are also cases where worker/resource problems cause confusing behavior: jobs fail to start pods or hit OOMs but parts of the system still mark workflows as completed or “success” from Temporal’s perspective. [[Resource leak](https://discuss.airbyte.io/t/2309); [Replication failure origin](https://discuss.airbyte.io/t/3619)] Why it can look “successful”: - The Temporal workflow driving the job may complete its control-flow steps (e.g., finishing the workload-launcher pipeline), so the high-level job status may be “succeeded”, even though the replication pod never actually ran or ran only partially. - The log line you pasted is from the platform / workload-launcher layer, not necessarily the final job status computation. What you should check / do next: 1. Look at the replication pod for that job: -
kubectl get pods -n <namespace>
and find the
replication-job-<job>-attempt-<n>
pod. -
kubectl describe pod ...
and check Events for “Insufficient cpu/memory”, OOMKilled, or scheduling failures. Similar symptoms have been reported and fixed by increasing CPU on the node. [[Mysql issue](https://github.com/airbytehq/airbyte/issues/68104)] 2. Confirm actual job outcome in Airbyte UI / API: - Open the job in the Airbyte UI and confirm: - Was any data actually written? - Are there retry attempts or partial-success logs? 3. Check and adjust resource settings: - On Kubernetes / Helm, ensure your workload / replication resource requests and limits are sane, and not conflicting with any per-connector overrides in the DB. There is a known issue where actor-specific resource settings can override your global
workloads.resources
and cause mismatches like “requested 2Gi while limit is 500Mi”. [[workload resources](https://github.com/airbytehq/airbyte/issues/68162)] - If you see pods failing to start due to insufficient resources, either: - Increase node capacity (more vCPUs / memory), or - Lower the per-job requests/limits, or - Run fewer concurrent syncs. 4. Look for repeated ResourceConstraintException messages: - If this happens regularly, it’s an indication your cluster is under-provisioned for the number/size of syncs you’re running. Because the knowledge sources don’t contain a specific note explaining why such jobs are always marked successful in this exact scenario, I can’t say definitively whether this is a UI/Temporal-status bug or expected edge behavior; but all available evidence points to resource exhaustion preventing the replication pod from starting while the control workflow still completes. If you can share: - The job ID, - The
kubectl describe
output of the replication pod for that job, - Your Helm
workloads.resources
(or job resources) settings, then you can compare them directly with the patterns in the issues above to confirm it’s the same resource-constraint situation.
i
@kapa.ai i have autoscaling setup in my cluster. The sync that is failing has these limits
Copy code
'{"cpu_limit": "3", "cpu_request": "2", "memory_limit": "4Gi", "memory_request": "3Gi"}'
Why would it not schedule on a new node if not availbel on current ones ?
k
A message from kapa.ai
In Airbyte’s model, this error is raised by the workload launcher when it cannot start the REPLICATION pod due to Kubernetes resource constraints; it does not control whether the cluster autoscaler adds nodes or where pods are scheduled. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)] From the available docs/issues, we can say: - Airbyte expects that “scaling Airbyte is a matter of ensuring that the Kubernetes cluster … has sufficient resources to schedule its various job pods,” but it does not include any special logic to force autoscaling; it simply asks K8s to create a pod and fails if that pod cannot be started. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)] - There is an open issue where the workload launcher reports resource errors like
Unable to start the REPLICATION pod
or “Invalid resource requirements detected” when limits/requests for the actor are inconsistent or overridden by existing DB values, regardless of cluster autoscaling. [[workload resources issue](https://github.com/airbytehq/airbyte/issues/68162)] The knowledge sources do not explain Kubernetes autoscaler behavior (e.g., when it chooses to add nodes, pod priority, taints, PDBs, unschedulable pods, etc.), nor how your specific autoscaling policy is configured. They only state that: - You must ensure the K8s cluster can schedule up to
2 x <number-of-possible-concurrent-connections>
pods. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes)] - Resource constraints at the cluster level can prevent pods from starting, causing resource-related job failures. [[Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#connector-cpu--memory-settings)] So, based on the docs: - Airbyte is behaving as designed: it tries to launch a REPLICATION pod with your specified
cpu_request=2
,
cpu_limit=3
,
memory_request=3Gi
,
memory_limit=4Gi
. - If Kubernetes (plus your cluster autoscaler configuration) does not schedule or scale out to satisfy that pod, Airbyte will see it as a resource failure and raise
ResourceConstraintException
. - The documentation does not say why Kubernetes/your autoscaler would not add a new node for such a pod; that depends on your K8s / autoscaler settings and is outside what the Airbyte docs cover. To move forward, the Airbyte-side actions the docs recommend are: 1. Verify that the cluster in practice can handle the total number of concurrent pods and their requests; otherwise, reduce concurrent syncs or connector resource requests. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes); [Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)] 2. Check for per-connector DB overrides that might be setting different, possibly invalid resource requirements compared to your global settings. [[workload resources issue](https://github.com/airbytehq/airbyte/issues/68162)] Anything beyond that (e.g., “why exactly didn’t the autoscaler create a node?”) is not described in the Airbyte knowledge sources, so I can’t answer that part confidently from this material alone.
i
@kapa.ai can i configure resources for source, destiantion and main sync job separately ?
k
A message from kapa.ai
Yes, you can configure resources with different scopes, but not as three independent knobs called “source”, “destination”, and “main sync job” in the way you might expect. From the docs: 1. Configuration levels (for sync jobs) [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources)] - Instance‑wide: via env vars /
values.yaml
, e.g.
JOB_MAIN_CONTAINER_CPU_REQUEST
,
JOB_MAIN_CONTAINER_CPU_LIMIT
,
JOB_MAIN_CONTAINER_MEMORY_REQUEST
,
JOB_MAIN_CONTAINER_MEMORY_LIMIT
. These apply to all job containers unless overridden. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-instance-wide-requirements)] - Connector definition: all sources/destinations of a given type (e.g. all MongoDB sources) via the
actor_definition.resource_requirements
field in the DB. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-definition-requirements)] - Connector‑specific: one specific connector instance (e.g. one MongoDB source) — available via UI/API for Self‑Managed Enterprise only. [[Configuring connector‑specific requirements](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-specific-requirements)] - Connection‑specific: one specific source–destination pair (the connection). Precedence (highest wins): Connection > Connector > Connector definition > Instance‑wide. [[1.5 release notes](https://docs.airbyte.com/release_notes/v-1.5#-self-managed-enterprise-changes)] 2. Per‑connector (source vs destination) Because connector‑specific and connector‑definition configs are tied to a specific actor (source or destination), you can give: - one resource profile to the source connector, and - a different profile to the destination connector, for the same connection. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources)] In Enterprise, you do this via the “Connector resource allocation” dropdown on each connector’s Settings tab or via API. [[Configuring connector‑specific requirements](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-specific-requirements)] 3. “Main sync job” / orchestrator vs read/write containers Instance‑wide env vars (
JOB_MAIN_CONTAINER_*
) apply to job containers, including the orchestrator pods. [[Orchestrator limits](https://discuss.airbyte.io/t/3775)] The docs and forum answer explicitly say the orchestrator gets its resource requirements from
JOB_MAIN_CONTAINER_*
and there is no separate documented set of env vars to configure orchestrator vs read/write differently. [[Orchestrator limits](https://discuss.airbyte.io/t/3775)] So, in practice, with what’s documented: - You can: - Set different resources for source vs destination (per connector or connector definition, Enterprise gives a UI for this). - Override per connection when needed. - Control global defaults for all job containers with
JOB_MAIN_CONTAINER_*
. - You cannot (per docs): - Independently tune “main sync job/orchestrator” vs “read” vs “write” containers with three distinct resource knobs; orchestrator uses the same
JOB_MAIN_CONTAINER_*
settings. [[Orchestrator limits](https://discuss.airbyte.io/t/3775)] If you’re on Self‑Managed Community/Core, connector‑specific UI controls are not available; you’re limited to instance‑wide env vars and DB‑level `actor_definition.resource_requirements`/connection overrides. [[Connector‑specific requirements](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-specific-requirements)]
i
@kapa.ai how to do this in the helm values ?
k
A message from kapa.ai
In the Helm chart, the only instance‑wide knobs that docs expose are the
JOB_MAIN_CONTAINER_*
env vars, set under
global.env_vars
in
values.yaml
(Helm v1) or equivalent. These apply to all job containers (orchestrator + read/write) unless overridden by DB‑level settings. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-instance-wide-requirements)] Example `values.yaml`:
Copy code
yaml global:   env_vars:     JOB_MAIN_CONTAINER_CPU_REQUEST: "2"     JOB_MAIN_CONTAINER_CPU_LIMIT: "3"     JOB_MAIN_CONTAINER_MEMORY_REQUEST: "3Gi"     JOB_MAIN_CONTAINER_MEMORY_LIMIT: "4Gi"
This is the documented way to configure instance‑wide sync job resources when deploying via Helm. [[OOM issue example](https://github.com/airbytehq/airbyte/issues/48816)] To set different resources for source vs destination vs a particular connection, you cannot do that purely in Helm; you must use the DB‑backed resource requirement fields (
actor_definition.resource_requirements
, connector‑specific, or connection‑specific overrides) as described in the connector resources guide. [[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-definition-requirements)] The docs do not provide Helm values for separately tuning orchestrator/read/write or per‑connector resources.
i
@kapa.ai what does this exactly mean ? To be safe, make sure the Kubernetes cluster can schedule up to
2 x <number-of-possible-concurrent-connections>
pods at once. This is the worse case estimate, and most users should be fine with
2 x <number-of-possible-concurrent-connections>
as a rule of thumb.
k
A message from kapa.ai
That line is giving you a capacity planning rule of thumb for Kubernetes when running Airbyte. From the docs: - Each concurrent sync uses two workers: > “Sync jobs use two workers. One worker reads from the source; the other worker writes to the destination.” [[What to scale](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)] - Therefore, if you can have N syncs running at the same time, you should assume up to 2 × N pods for those syncs (one pod per worker) may need to be scheduled concurrently. So: - If your maximum “possible concurrent connections” is 50, plan your cluster so it can schedule roughly 100 pods at once for Airbyte syncs. - This is a worst‑case estimate / safety margin; many real workloads will use fewer pods, but sizing for
2 × concurrent_syncs
helps avoid unschedulable pods and resource errors. [[What to scale](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)]
i
@kapa.ai on aws what kind of instance i should start for airbyte
k
A message from kapa.ai
On AWS, the documented guidance depends on how you deploy Airbyte: ### 1. Self-Managed Enterprise on Kubernetes (EKS) For a production-ready Self-Managed Enterprise deployment on AWS, Airbyte recommends: - Kubernetes cluster: Amazon EKS on EC2, across 2+ Availability Zones, with a minimum of 6 nodes. - Instance type: Use memory‑optimized instances, such as M7i / M7g families. - Per‑node size: At least 2 vCPUs and 8 GiB RAM. [[Enterprise prerequisites](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#infrastructure-prerequisites)] - Disk: At least 30 GiB disk per node, and preferably more (storage is cheap). [[What to scale](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#disk-space)] In general, Airbyte recommends starting with a mid‑sized instance (4–8 cores) and then tuning up or down based on workload. [[What to scale](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)] ### 2. Single‑node EC2 (non‑Kubernetes) From the forum and EC2 deployment docs (linked in the forum post), for a single EC2 instance: - t3.micro is not recommended; it led to severe slowness and freezing. [[EC2 slow t3.micro](https://discuss.airbyte.io/t/2334)] - For testing: t2.medium is likely sufficient. - For production: t2.large or better is recommended. [[EC2 slow t3.micro](https://discuss.airbyte.io/t/2334)] If you share whether you’re planning EKS (cluster) or a single EC2 VM, you can pick from the above accordingly.