Hi everyone 👋 I’m looking for some help troubleshooting long-running syncs and tuning for Airbyte v2. We’re running Airbyte on AWS with Kubernetes, and some HubSpot connections remain in the running state for several hours during syncs. Our goal is to achieve maximum possible performance and understand whether this behavior is expected or can be improved with better configuration. Current worker configuration:

resources:
  requests:
    memory: "2Gi"
    cpu: "1"
  limits:
    memory: "4Gi"
    cpu: "2"

I’d appreciate guidance on: • Any Airbyte v2–specific settings or flags that impact replication performance • Best practices around HubSpot rate limits, stream selection, or parallelism • Kubernetes patterns or misconfigurations that could lead to very long-running syncs Any advice, docs, or shared experiences would be really helpful. Thanks in advance to the community for the support 🙏

You’re running into a few different (but related) dimensions: Airbyte concurrency limits, Kubernetes capacity, and HubSpot’s own API/rate‑limit behavior. The knowledge sources don’t mention Airbyte v2–only flags in detail, but they do cover how scaling, concurrency, and HubSpot performance interact; I’ll stick to those and call out where things are not explicitly documented. --- ### 1. Airbyte (v2) concurrency & scaling knobs On Kubernetes, the main levers for how many syncs can run at once are: - Worker concurrency limits (per worker pod) - Number of worker replicas (how many worker pods) - Cluster capacity (can K8s actually schedule all the job pods?) For self‑managed / Enterprise, the docs describe: - Each concurrent sync needs at least 3 pods:

orchestrator

,

read

,

write

. So

N

concurrent syncs ≈

3N

extra pods in your namespace. [[Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)] - To avoid stuck jobs, you can cap concurrent syncs via worker settings:

yaml   # Helm chart v2 style   worker:     maxSyncWorkers: 5        # max concurrent syncs per worker     maxCheckWorkers: 5     replicaCount: 2          # number of worker pods

or via env vars in Helm v1: [[Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits)]

yaml   worker:     extraEnv:       - name: MAX_SYNC_WORKERS         value: "5"       - name: MAX_CHECK_WORKERS         value: "5"

- Effective max concurrent syncs ≈

worker.replicaCount * MAX_SYNC_WORKERS

. [[How to configure concurrency](https://discuss.airbyte.io/t/1077)] If you’re seeing jobs “stuck in running” with otherwise healthy CPU/memory, it can be a symptom of: - Too many concurrent syncs vs. cluster capacity (pods can’t be scheduled, jobs appear to hang). [[Scaling – K8s](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes)] - Over‑aggressive concurrency settings (very high

MAX_SYNC_WORKERS

and many replicas) leading to thousands of active jobs that don’t complete cleanly. [[How to configure concurrency](https://discuss.airbyte.io/t/1077)] Actionable suggestions (from the docs & forum examples): - Start with moderate values and scale up: - `worker.replicaCount`: 2–3 -

MAX_SYNC_WORKERS

/ `worker.maxSyncWorkers`: 5–15 - Ensure your K8s cluster can schedule at least

2 × <number-of-possible-concurrent-connections>

pods as a rule of thumb. [[Scaling – K8s](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes)] --- ### 2. HubSpot‑specific behavior & expectations The knowledge sources show that long HubSpot syncs are common and often driven by HubSpot’s own API constraints: - Large HubSpot streams (e.g.,

email_events

,

contacts

) can run for many hours due to API rate limits and pagination behavior. [[HubSpot performance](https://discuss.airbyte.io/t/2654); [HubSpot contacts perf](https://discuss.airbyte.io/t/2219)] - For very large volumes (millions of records), the connector advances its bookmark in small time slices (seconds at a time), which can look “slow but progressing” in logs. [[HubSpot contacts perf](https://discuss.airbyte.io/t/2219)] From the docs and forum threads, practical patterns are: - Limit backfill scope: - Use a more recent

start_date

for initial tests to confirm behavior and isolate whether the slowness is just volume/API‑limit related. [[HubSpot performance](https://discuss.airbyte.io/t/2654)] - Stream selection: - Only enable the HubSpot streams you truly need; some (like

email_events

,

contact_list_membership

) are extremely heavy and can dominate runtime. [[HubSpot performance](https://discuss.airbyte.io/t/2654); [HubSpot contacts perf](https://discuss.airbyte.io/t/2219)] - Mode choice: - For very large historical loads, some users consider doing a one‑time full refresh and then switching to incremental; the docs don’t prescribe this explicitly, but the performance issues described for incremental backfills over many months/years suggest that a staged approach can be helpful. [[HubSpot contacts perf](https://discuss.airbyte.io/t/2219)] Given this, multi‑hour HubSpot syncs can be expected when: - You’re backfilling many months/years of data, and/or - You have high‑volume streams enabled, and - HubSpot’s API rate limits are the bottleneck. --- ### 3. Kubernetes resource patterns & “stuck” / long‑running jobs The docs and forum posts highlight a few K8s‑related causes of long‑running or stuck syncs: 1. Insufficient cluster capacity / scheduling delays - If the cluster can’t schedule all required pods (due to CPU/memory requests or node limits), jobs can sit in “running” while nothing progresses. [[Scaling – K8s](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes); [Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)] - Symptoms: jobs stuck in

Running

for hours, low actual CPU usage, nodes scaling down while many jobs are “running”. [[How to configure concurrency](https://discuss.airbyte.io/t/1077); [All syncs hanging](https://discuss.airbyte.io/t/1916)] 2. Under‑provisioned job resources - If connector pods don’t have enough memory/CPU, you can see: - Jobs stuck in

Running

or

Retrying

- Occasional “stuck jobs” that only clear after pod restarts. [[Sync stuck in running](https://discuss.airbyte.io/t/680); [Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#connector-cpu--memory-settings)] - Docs recommend: - Monitoring requested vs. used CPU/memory - Increasing job limits/requests if jobs are starved. [[Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#connector-cpu--memory-settings)] 3. Too much concurrency vs. Temporal / infra - Very high concurrency (thousands of jobs) can lead to jobs that are technically “done” but remain in

Running

due to orchestration issues (e.g., stuck at “Setting kube job …” for hours). [[How to configure concurrency](https://discuss.airbyte.io/t/1077)] - In those tests, reducing concurrency (workers and ports) was necessary to get jobs to complete gracefully. [[How to configure concurrency](https://discuss.airbyte.io/t/1077)] Your current per‑job resources:

yaml requests:   memory: "2Gi"   cpu: "1" limits:   memory: "4Gi"   cpu: "2"

are in line with what the docs call “mid‑sized” and should be fine for many HubSpot syncs, but the effective behavior depends on: - How many syncs you allow concurrently (via

maxSyncWorkers

and replicas) - Whether your cluster has enough nodes to schedule

3 × concurrent_syncs

pods with those requests --- ### 4. Concrete tuning steps to try Based on the docs and forum content: 1. Right‑size concurrency - Set explicit limits: - `worker.replicaCount`: start with 2–3 -

worker.maxSyncWorkers

(or

MAX_SYNC_WORKERS

): start with 5–10 - Verify that

2 × (replicaCount × maxSyncWorkers)

is within what your cluster can comfortably schedule. [[Scaling – K8s](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes); [Enterprise scaling](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits)] 2. Validate cluster capacity - Use K8s metrics to confirm: - Pods for

orchestrator

,

read

,

write

are being scheduled quickly - Nodes are not constantly scaling down while many jobs are “running” (a pattern seen in high‑concurrency tests). [[How to configure concurrency](https://discuss.airbyte.io/t/1077)] 3. HubSpot‑specific hygiene - For each HubSpot connection: - Limit streams to what you need (especially avoid or isolate very heavy ones like

email_events

or

contact_list_membership

when testing). [[HubSpot performance](https://discuss.airbyte.io/t/2654); [HubSpot contacts perf](https://discuss.airbyte.io/t/2219)] - For backfills, test with a recent

start_date

to confirm that short‑range syncs complete quickly; if they do, long runtimes on older ranges are likely just API/volume‑driven. [[HubSpot performance](https://discuss.airbyte.io/t/2654)] 4. Watch for “stuck” vs. “slow but progressing” - If logs show bookmarks/time windows advancing (even slowly), that’s usually HubSpot/API‑bound behavior. [[HubSpot contacts perf](https://discuss.airbyte.io/t/2219)] - If logs stop after orchestration messages (e.g., only “Executing worker wrapper” or “Setting kube job …” for hours), that points more to orchestration / K8s / resource issues. [[Sync stuck in running](https://discuss.airbyte.io/t/680); [How to configure concurrency](https://discuss.airbyte.io/t/1077); [All syncs hanging](https://discuss.airbyte.io/t/1916)] --- Because the knowledge sources don’t list Airbyte v2–only flags beyond these general scaling and concurrency settings, I can’t point you to additional v2‑specific toggles with confidence. If you can share: - How many worker replicas you’re running - Your

maxSyncWorkers

/

MAX_SYNC_WORKERS

values - Rough number of concurrent HubSpot syncs I can help map that to the documented patterns above and suggest a more concrete target configuration.

also worth double-checking that your resource limits are actually being applied. certain HubSpot streams, like form submissions, aren't incremental by default (you have to manually select the columns to use as a primary key)

@Justin Beasley thank you for your message, we're just now trying to do the initial sync. I'll keep in mind what you mentioned for the incremental sync