@kapa.ai hi i upgraded from airbyte version 1.4.0 to 1.7.8 and now i see only 15 replication pods (parallel syncs) in my k8

but i have MAX_SYNC_WORKERS = 75

config example

worker:
      replicaCount: 3
      resources:
        requests:
          cpu: 1000m
          memory: 4Gi
        limits:
          cpu: 2000m
          memory: 4Gi
      extraEnv:
        - name: MAX_SYNC_WORKERS
          value: "75"
        - name: MAX_CHECK_WORKERS
          value: "20"
      affinity: *airbyte_affinity

why i have max 15 in real app run

The knowledge sources don’t explain a behavior change specific to upgrading from 1.4.0 to 1.7.8, but they do clarify how concurrency is actually limited, which likely explains why you only see ~15 parallel syncs. Key points from the docs: 1.

MAX_SYNC_WORKERS

is per worker pod, not a global limit It “defines the maximum number of Sync workers each Airbyte Worker container can support.”[[Configuring Airbyte](https://docs.airbyte.com/platform/1.7/operator-guides/configuring-airbyte#worker)] With

replicaCount: 3

and

MAX_SYNC_WORKERS=75

, the theoretical upper bound is 3 × 75 = 225 sync workers, but other limits apply. 2. There is also a global concurrency cap and port-based limits Older docs and forum posts describe the effective parallel syncs as constrained by: - A global scheduler cap (

SUBMITTER_NUM_THREADS

/ job submitter threads in older versions). - `TEMPORAL_WORKER_PORTS`: each worker can only run as many jobs as there are ports available; in k8s deployments, one worker is effectively limited to 40 parallel jobs by default.[[Scaling on k8s](https://discuss.airbyte.io/t/826); [Configuring Airbyte](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker)] Example from the forum: to run 100 sync jobs they recommend: - 3 workers -

MAX_SYNC_WORKERS = 100

-

SUBMITTER_NUM_THREADS = 200

and note that a single worker is limited by

TEMPORAL_WORKER_PORTS

.[[Scaling on k8s](https://discuss.airbyte.io/t/826)] 3. Enterprise scaling docs emphasize that

MAX_SYNC_WORKERS

is only one of several controls They show setting

MAX_SYNC_WORKERS

and

MAX_CHECK_WORKERS

via

worker.extraEnv

, and separately scaling

worker.replicaCount

.[[Scaling Airbyte 1.7](https://docs.airbyte.com/platform/1.7/enterprise-setup/scaling-airbyte#concurrent-sync-limits)] They do not guarantee that raising

MAX_SYNC_WORKERS

alone will increase actual parallel syncs. Given that: - Your

MAX_SYNC_WORKERS=75

is correctly set via

worker.extraEnv

for 1.7.x.[[Scaling Airbyte 1.7](https://docs.airbyte.com/platform/1.7/enterprise-setup/scaling-airbyte#concurrent-sync-limits)] - But the observed concurrency (~15) is likely being capped by: - The global scheduler / submitter concurrency (not shown in your snippet). - Or the effective limit from

TEMPORAL_WORKER_PORTS

and how many ports are actually exposed in your Helm deployment (default 9001–9040 = 40 ports per worker).[[Configuring Airbyte](https://docs.airbyte.com/platform/1.7/operator-guides/configuring-airbyte#worker)] The knowledge sources do not provide an exact formula for 1.7.8 or a documented “15” cap, so I can’t say precisely why you see 15 instead of, say, 30 or 40. But they do indicate that: -

MAX_SYNC_WORKERS

alone does not guarantee that many parallel syncs. - You must also ensure: -

TEMPORAL_WORKER_PORTS

range is large enough. - Any global scheduler/job-submitter concurrency setting is high enough. What to check in your deployment (based on docs): 1. In your Helm values, confirm

TEMPORAL_WORKER_PORTS

(or equivalent) is set to a sufficiently large range; default is 9001–9040 (40 ports).[[Configuring Airbyte](https://docs.airbyte.com/platform/1.7/operator-guides/configuring-airbyte#worker)] 2. Check if you have any global concurrency-related env vars still set from 1.4.0 (e.g.

SUBMITTER_NUM_THREADS

in older setups) that might be capping total jobs.[[Scaling on k8s](https://discuss.airbyte.io/t/826)] 3. Verify that all 3 worker pods are actually running and healthy, and that you’re counting only sync jobs (not checks/specs). If you can share your full worker-related Helm values (including any

TEMPORAL_WORKER_PORTS

or scheduler settings), you can compare them against the patterns in the scaling docs to see where the effective cap is coming from.

which default value for MAX_SYNC_WORKERS? @kapa.ai

A message from kapa.ai