<@U04S8PLJWMT> All my syncs are failing due to OOM...
# replication-ask-ai
j
@kapa.ai All my syncs are failing due to OOMKilled. The sync never ends. I tried changing launcher configuration launcher env confirmed 4Gi/8Gi Runtime pod still gets 1Gi/2Gi what am i missing ?
k
This is a known bug affecting multiple users. The resource configuration set via Helm values, environment variables, or ConfigMaps is not being properly propagated to the actual replication pods by the workload-launcher. [Resource limits ignored] [Orchestrator ignores limits] Root Cause The workload-launcher appears to use its own internal defaults instead of reading from the configured values. Even with
useConnectorResourceDefaults: false
, persisted actor definitions in the database may take precedence over Helm values and environment variables. [Resource limits ignored] Available Workarounds 1. Set resource overrides directly at the connection level via SQL
Copy code
UPDATE connection
SET resource_requirements = jsonb_build_object(
  'cpu_request', '0.5',
  'cpu_limit', '2',
  'memory_request', '1Gi',
  'memory_limit', '4Gi'
)
WHERE id = '<your-connection-id>';
> ⚠️ Note: This applies to source/destination containers. There is currently no direct override for the orchestrator container via this method. [Orchestrator ignores limits] 2. Set env vars directly on the workload-launcher deployment (not just ConfigMap)
Copy code
workloadLauncher:
  extraEnv:
    - name: JOB_MAIN_CONTAINER_CPU_LIMIT
      value: "4"
    - name: JOB_MAIN_CONTAINER_CPU_REQUEST
      value: "2"
    - name: JOB_MAIN_CONTAINER_MEMORY_LIMIT
      value: "8Gi"
    - name: JOB_MAIN_CONTAINER_MEMORY_REQUEST
      value: "4Gi"
Verify the values are actually visible to the pod:
Copy code
kubectl exec -n airbyte deploy/airbyte-workload-launcher -- env | grep JOB_MAIN
3. Kubernetes Mutating Webhook (for orchestrator container specifically) If the orchestrator container is stuck at 1Gi, a
MutatingWebhookConfiguration
can intercept pod CREATE requests and patch the
orchestrator
container's memory. It targets pods with label
airbyte=job-pod
. This has been confirmed to work by multiple users. [Orchestrator ignores limits] Status This is an open bug tracked internally by the Airbyte engineering team. There is no native fix available yet. The issues are tracked at: • #72833 — orchestrator memory always 1Gi • #73346 — resource limits ignored by abctl/workload-launcher 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.