Hello! During replication in Airbyte I am periodically encountering the following errors: • Workload Heartbeat Error • Workload failed, source: workload-monitor-heartbeat •

Failed to connect to airbyte-workload-api-server-svc:8007 (Connection refused)

•

No exit code found. exitCode.txt does not exist

In the logs it looks like the sync crashes in the middle of the process, and the source container does not manage to write the

exitCode.txt

file before termination. The data size per sync is approximately: • 3.7M records emitted (~379 MB total) • 3 streams (around 1M, 1.58M and 1.126M records) • The sync runs for ~4 minutes before failing. ❓ Questions for Support Could you please help me understand the root cause and how to resolve this issue? 1. What could cause the Workload Heartbeat Error? 2. Specifically, why would the workload be unable to connect to

airbyte-workload-api-server-svc:8007

(Connection refused)? 3. Why does the workload container fail to create

exitCode.txt

? 4. This seems to indicate the container crashes abruptly rather than exiting gracefully. 5. Could this problem be related to insufficient resources (RAM/CPU) on the workload container or the workload-api-server? 6. If so, what are the recommended resource requirements for handling large syncs (~4M records / ~400 MB)? 7. Are there any known issues in Airbyte where the Workload API Server loses connection during high-volume syncs? 8. What additional logs or metrics would you need from my environment to further diagnose the issue? 9. (pods logs, describe output, node metrics, etc.) 10. Do you have recommended approaches for optimizing large syncs? 11. For example: ◦ Increasing workload / heartbeat timeouts ◦ Tuning CPU/RAM for workloads or API server ◦ Splitting the sync into smaller batches ◦ Using the new Worker mode or other scaling strategies

Based on the shared docs and forum threads, here’s what we can say with confidence and where things are less clear. --- ### 1. What could cause the Workload Heartbeat Error? Heartbeats are triggered when Airbyte stops receiving messages from a source/destination for a configured period of time; this is meant as a catch‑all to detect unresponsive connectors and restart jobs. [[Heartbeats](https://docs.airbyte.com/platform/1.8/understanding-airbyte/heartbeats)] Known causes include: - Slow or stalled database queries (e.g., large tables, unindexed cursor columns, under‑provisioned DB). - Destinations being slow to accept writes (often due to limited resources). - API sources that rate‑limit or take a long time to respond. [[Heartbeats](https://docs.airbyte.com/platform/1.8/understanding-airbyte/heartbeats)] So a heartbeat error generally means: from the platform’s perspective, your source/destination stopped “talking” (no logs/messages) for too long. --- ### 2. Why would it fail to connect to

airbyte-workload-api-server-svc:8007

? The knowledge provided does not mention this specific service or port, so the exact failure mode is not documented. Given the pattern from other Kubernetes‑related issues, failures to talk to a control‑plane pod or service usually show up when: - The pod doesn’t exist or was garbage‑collected before Airbyte retrieves details (seen with GKE garbage collector in older versions). [[RuntimeException sync](https://discuss.airbyte.io/t/2421)] - There are resource or scaling issues causing pods to fail, not start, or be swept early when many jobs run concurrently. [[Too many syncs](https://discuss.airbyte.io/t/3856)] But there’s no explicit description of

airbyte-workload-api-server-svc

connection‑refused errors in the docs you shared, so the exact mechanism is not described. --- ### 3–4. Why does the workload container fail to create

exitCode.txt

/ abrupt crash? In the newer worker implementation, a “no exit code found / exitCode.txt does not exist” error is thrown when Airbyte can’t read the exit status from the container. This is analogous to older issues where: - Airbyte reported

Cannot find pod while trying to retrieve exit code

because the pod was gone before exit code retrieval. [[RuntimeException sync](https://discuss.airbyte.io/t/2421); [GH 56926](https://github.com/airbytehq/airbyte/issues/56926)] In one open issue, a long‑running sync to S3 data lake shows:

text Destination has timed out on accept call ... java.lang.IllegalStateException: No exit code found.

indicating the connector timed out (heartbeat / timeout monitor) and then the worker couldn’t get the exit code. [[GH 56926](https://github.com/airbytehq/airbyte/issues/56926)] So your observation is aligned: if the container is killed (OOM, node eviction, platform timeout, heartbeat timeout) instead of exiting cleanly,

exitCode.txt

may never be written and the worker reports “No exit code found.” --- ### 5–6. Could this be resource‑related? What are the recommendations? The docs make it clear that sync scalability is primarily about **memory and disk**; CPU matters but is less often the hard failure point. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)] Key points: - Source worker memory is usually the main culprit. A source can hold up to 10,000 records in memory; for large rows this can be several GB of RAM (e.g., 10k × 0.5 MB ≈ 5 GB). [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#memory)] - For long‑running or large syncs, under‑provisioned memory can cause crashes or OOM‑kills that look like abrupt terminations. [[Large sync fails](https://discuss.airbyte.io/t/782)] In one forum case, large DB jobs failing inconsistently were resolved by: - Increasing job memory requests (

jobs.resources.requests.memory: 1Gi

), and - Upgrading Airbyte and connectors (which included memory‑usage improvements). [[Large sync fails](https://discuss.airbyte.io/t/782)] The general recommendation from the scaling guide: - Start with a mid‑sized instance (4–8 cores) and then tune to workload. - Allocate at least 30 GB disk per node to avoid disk‑related issues. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#disk-space)] The docs do not give a precise “X GB for 4M records” formula, but they clearly recommend increasing connector/job memory when dealing with large row counts or large rows. Given your ~3.7M records / ~379 MB total and ~4‑minute run time, this is not huge in volume, but if individual rows are relatively large or there are multiple concurrent jobs, it is still plausible that memory or CPU pressure is causing container instability. --- ### 7. Known issues with long / high‑volume syncs and exit codes / timeouts From the provided sources: - Pods disappearing before exit code retrieval for long‑running syncs on GKE – fixed in earlier versions, but it shows that long jobs can surface platform edge cases. [[RuntimeException sync](https://discuss.airbyte.io/t/2421)] - S3 Data Lake destination: open issue where large/long‑running connections stall and then report

Destination has timed out on accept call

followed by

No exit code found.

[[GH 56926](https://github.com/airbytehq/airbyte/issues/56926)] These show that: - There are known situations where long/large syncs interact badly with the timeout/heartbeat/exit‑code handling and end in “No exit code found” rather than a clear connector error. The docs you shared do not explicitly document a known bug where “Workload API server loses connection during high‑volume syncs,” so if that’s happening, it’s not described here. --- ### 8–9. What extra logs/metrics would help? The knowledge sources don’t provide a support runbook, but from the issues and discussions, the following have been used to diagnose similar cases: - Pod logs for: - Source and destination containers. - Any sidecars / workload controller / worker pods. - Kubernetes events / describe: -

kubectl describe pod <source-pod>

and

<destination-pod>

to see if there were OOMKilled, Evicted, or Restarted statuses. - Cluster/node metrics: - Node memory and CPU usage during the sync, to detect pressure or eviction. - For Kubernetes installs: - Logs from worker/controller components that manage job pods (in older setups:

airbyte-worker

and container orchestrator; in newer workload‑based setups: anything related to workloads). This pattern is consistent across the forum posts where resource issues or Kubernetes garbage‑collection were suspected. [[Too many syncs](https://discuss.airbyte.io/t/3856); [RuntimeException sync](https://discuss.airbyte.io/t/2421); [Large sync fails](https://discuss.airbyte.io/t/782)] --- ### 10–11. Recommended approaches for optimizing / stabilizing large syncs From the scaling and heartbeat docs plus forum experiences: 1. Increase connector / job resources - Use the “Configuring connector resources” guidance (not included in your snippets, but referenced). [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#memory)] - Increase memory requests/limits for jobs, especially the source worker and any large‑load destinations. - In Kubernetes deployments, ensure your nodes have enough RAM and disk and can schedule

2 × <concurrent-connections>

pods. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes)] 2. Check for DB‑side bottlenecks - For database sources, ensure cursor columns are indexed and the source DB is not under‑provisioned. Slow queries can trigger heartbeat timeouts. [[Heartbeats](https://docs.airbyte.com/platform/1.8/understanding-airbyte/heartbeats)] 3. Upgrade Airbyte and connectors - Several forum threads show long‑running sync issues improved after upgrading Airbyte and connectors (including memory‑usage optimizations and fixes in exit‑code/pod handling). [[Large sync fails](https://discuss.airbyte.io/t/782); [RuntimeException sync](https://discuss.airbyte.io/t/2421)] 4. Limit concurrency / avoid overload - In one case, running many syncs concurrently led to non‑explicit failures involving Kubernetes issues (

kubectl cp failed

, port allocation problems). Reducing concurrency or increasing cluster capacity was needed. [[Too many syncs](https://discuss.airbyte.io/t/3856)] - Ensure your cluster can handle

2 × <max concurrent connections>

pods. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#on-kubernetes)] 5. Consider breaking up very large initial loads - While not explicitly documented as an official strategy, forum users often test smaller subsets first and see that small jobs succeed while large ones fail, which then directs them to resource tuning. [[Large sync fails](https://discuss.airbyte.io/t/782); [RuntimeException sync](https://discuss.airbyte.io/t/2421)] 6. Timeout / heartbeat configuration - The docs explain what heartbeats are and known causes, but the specific knobs and recommended values for increasing heartbeat or timeout thresholds are not provided in the snippets you shared. [[Heartbeats](https://docs.airbyte.com/platform/1.8/understanding-airbyte/heartbeats)] - So: it is conceptually valid that tuning these could help if the connector is slow but healthy, but I cannot point you to exact parameters or values from the provided material. 7. Worker mode / newer workload system - The scaling guide discusses the workload system and how each job is a pod, but does not provide detailed migration/“worker mode” recommendations in the excerpts you provided. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)] - So I can’t reliably advise on “new Worker mode” specifics from these docs alone. --- ### Summary - The Workload Heartbeat Error and

No exit code found / exitCode.txt does not exist

together point to the connector container becoming unresponsive or being killed before it can exit cleanly, similar to documented issues around long‑running syncs and exit‑code retrieval. [[Heartbeats](https://docs.airbyte.com/platform/1.8/understanding-airbyte/heartbeats); [GH 56926](https://github.com/airbytehq/airbyte/issues/56926); [RuntimeException sync](https://discuss.airbyte.io/t/2421)] - Resource constraints (especially memory) and slow DB/destination performance are common underlying causes for such behavior in large/long‑running syncs. [[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#memory); [Large sync fails](https://discuss.airbyte.io/t/782)] - There are known issues where long runs interact poorly with exit‑code handling; one is still open for S3 data lake. [[GH 56926](https://github.com/airbytehq/airbyte/issues/56926)] - For deeper diagnosis in your environment, Kubernetes pod logs (source/destination and workload components),

kubectl describe

output, and node metrics around memory/CPU during the sync are the most immediately useful artifacts, as illustrated in similar forum threads. [[Too many syncs](https://discuss.airbyte.io/t/3856); [Large sync fails](https://discuss.airbyte.io/t/782)] If you can share (or review internally) those pod

describe

outputs and node metrics, focus especially on any OOMKilled/Evicted statuses or spikes in memory around the 4‑minute mark when the sync fails.