@kapa.ai how can I make the airbyte sync have more workers to do thing in parralel

To run more things in parallel in Airbyte you need to increase the concurrency limits for workers and, if needed, the number of worker pods. There are two main levers: 1. Increase max concurrent workers Set these environment variables on the worker service:

worker:
  # Helm v1 style
  extraEnv:
    - name: MAX_SYNC_WORKERS
      value: "10"   # or higher, e.g. 20
    - name: MAX_CHECK_WORKERS
      value: "10"

Or with newer Helm charts (v2):

worker:
  maxSyncWorkers: 10
  maxCheckWorkers: 10

These control how many sync/check jobs a single worker can run in parallel. Defaults are 5. [Scaling Airbyte][Scaling 2.0] Older docs and forum posts also mention

MAX_SPEC_WORKERS

and

MAX_DISCOVERY_WORKERS

for spec/discover jobs. [Configuring jobs][Parallel streams] 2. Increase number of worker replicas If you want even more parallel syncs, scale out the worker deployment:

worker:
  replicaCount: 3   # or more

Each worker pod can run up to

MAX_SYNC_WORKERS

syncs concurrently, so total possible concurrent syncs ≈

replicaCount * MAX_SYNC_WORKERS

. [Scaling Airbyte] 3. Be mindful of cluster resources More parallel workers mean more connector pods (

orchestrator

,

read

,

write

) and higher CPU/memory usage. If you see “stuck” jobs or scheduling issues, you may need to: • Increase node capacity, or • Lower per-job resource requests/limits in

global.jobs.resources

so pods can be scheduled. [Scaling Airbyte] If you share how you deployed Airbyte (Docker vs Helm, version), I can point to the exact place to change these values. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai But i mean the sync jobs, the replication pods, I want to scale those

To scale the replication (sync) pods themselves, you control how many sync jobs can run concurrently across your deployment. From the docs and forum posts, the levers are: 1. Limit / increase concurrent syncs Set

MAX_SYNC_WORKERS

on the worker service. This is the max number of sync jobs a worker can run in parallel, which directly controls how many replication pods can be active at once. [Scaling Airbyte][Parallel streams] Helm v1 style:

worker:
  extraEnv:
    - name: MAX_SYNC_WORKERS
      value: "10"   # increase from default 5
    - name: MAX_CHECK_WORKERS
      value: "10"

Helm v2 style: [Scaling 1.8][Scaling 2.0]

worker:
  maxSyncWorkers: 10
  maxCheckWorkers: 10

Total potential concurrent replication pods ≈

worker.replicaCount * MAX_SYNC_WORKERS

. 2. Increase worker replicas Each sync job creates at least 3 connector pods (

orchestrator

,

read

,

write

). To run more of them in parallel, increase the number of worker pods: [Scaling Airbyte]

worker:
  replicaCount: 3   # or higher

With

replicaCount: 3

and

maxSyncWorkers: 10

, you can have up to ~30 concurrent syncs (so ~90 connector pods). 3. Ensure cluster can schedule the extra pods Because each concurrent sync adds multiple pods, you may need to: • Add more nodes, or • Reduce per-job CPU/memory requests in

global.jobs.resources

so Kubernetes can schedule more replication pods. [Scaling Airbyte] If you tell me your Airbyte version and whether you use Helm v1 or v2, I can show the exact

values.yaml

snippet for your case. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.