@kapa.ai Are there any values.yml, or otherwise, options that I can set to limit the overhead cost of source and destination containers spinning up. Maybe some optional checks or otherwise that I can turn off.

Yes, there are several options you can configure to reduce the overhead of source and destination connector containers. 1. Set Resource Limits for Connector Pods (Instance-Wide) You can control the CPU and memory allocated to all connector job containers via

values.yaml

. This prevents over-provisioning and reduces resource overhead:

global:
  jobs:
    resources:
      limits:
        cpu: 250m       # e.g. reduce from defaults
        memory: 500m
      requests:
        cpu: 75m
        memory: 150m

[Scaling Airbyte] These map to the environment variables

JOB_MAIN_CONTAINER_CPU_REQUEST

,

JOB_MAIN_CONTAINER_CPU_LIMIT

,

JOB_MAIN_CONTAINER_MEMORY_REQUEST

, and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

. [Configuring Connector Resources] 2. Limit Concurrent Syncs Each concurrent sync spins up at least 3 additional connector pods (

orchestrator

,

read

,

write

). Reducing the number of concurrent syncs directly reduces overhead:

worker:
  maxSyncWorkers: 5    # limit concurrent syncs
  maxCheckWorkers: 5   # limit concurrent checks

[Scaling Airbyte] 3. Fine-Grained Resource Control (Per Connector or Connection) If only specific connectors are resource-heavy, you can override resources at the connector definition level via SQL rather than raising limits globally:

update actor_definition 
set resource_requirements = '{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "2", "cpu_request": "2", "memory_limit": "2048Mi", "memory_request": "2048Mi"}}]}' 
where id = '<id-from-step-1>';

This avoids over-allocating resources to connectors that don't need them. [Configuring Connector Resources] Note: There is a known issue where

global.jobs.resources

settings in

values.yaml

may be ignored in some versions (e.g., 1.2.0 and 2.0.x), with pods defaulting to hardcoded resource values. [OOM Issue; Resource Limits Ignored] If you observe this, verify that the settings are actually being applied to spawned pods using

kubectl describe pod

. The knowledge sources do not explicitly mention an option to disable connection checks (e.g., the pre-sync source/destination validation) via

values.yaml

. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.