Dilyan Stoev
08/24/2026, 7:20 AM2026-08-24 07:07:35,275 [main] INFO i.a.c.ConnectorMessageProcessor(updateConfigFromControlMessage):280 - Checking for optional control message...
2026-08-24 07:07:35,402 [main] INFO i.a.c.ConnectorWatcher(saveConnectorOutput):202 - Writing output of ec27084b-13f2-4883-acc5-57e11896249f_84_0_check to the doc store
2026-08-24 07:07:36,038 [pool-3-thread-1] WARN i.a.c.HeartbeatMonitor$HeartbeatTask(handleException):112 - Cancelling job, workload is in a terminal state
io.airbyte.api.client.ApiException: Gone
at io.airbyte.workload.api.client.WorkloadApiClient.unit(WorkloadApiClient.kt:417)
at io.airbyte.workload.api.client.WorkloadApiClient.workloadRunning(WorkloadApiClient.kt:58)
at io.airbyte.connectorSidecar.HeartbeatMonitor$HeartbeatTask.run(HeartbeatMonitor.kt:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
2026-08-24 07:07:36,690 [main] INFO i.a.c.ConnectorWatcher(markWorkloadSuccess):207 - Marking workload ec27084b-13f2-4883-acc5-57e11896249f_84_0_check as successful
2026-08-24 07:07:36,801 [main] ERROR i.a.c.ConnectorWatcher(handleException):216 - Error performing operation: io.airbyte.api.client.ApiException
io.airbyte.api.client.ApiException: Gone
at io.airbyte.workload.api.client.WorkloadApiClient.unit(WorkloadApiClient.kt:417)
at io.airbyte.workload.api.client.WorkloadApiClient.workloadSuccess(WorkloadApiClient.kt:56)
at io.airbyte.connectorSidecar.ConnectorWatcher.markWorkloadSuccess(ConnectorWatcher.kt:208)
at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:95)
at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:22)
at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt)
2026-08-24 07:07:37,254 [main] INFO i.a.c.ConnectorWatcher(failWorkload):359 - Failing workload ec27084b-13f2-4883-acc5-57e11896249f_84_0_check.
2026-08-24 07:07:37,451 [main] INFO i.a.c.ConnectorWatcher(failWorkload):359 - Failing workload ec27084b-13f2-4883-acc5-57e11896249f_84_0_check.
2026-08-24 07:07:37,471 [main] INFO i.a.c.ConnectorWatcher(exitInternalError):271 - Deliberately exiting process with code 1.
for pod source-redshift-check-84-0-dzexm .kapa.ai
08/24/2026, 7:20 AMDilyan Stoev
08/24/2026, 7:21 AM2026-08-24 07:07:35,275 [main] INFO i.a.c.ConnectorMessageProcessor(updateConfigFromControlMessage):280 - Checking for optional control message...
2026-08-24 07:07:35,402 [main] INFO i.a.c.ConnectorWatcher(saveConnectorOutput):202 - Writing output of ec27084b-13f2-4883-acc5-57e11896249f_84_0_check to the doc store
2026-08-24 07:07:36,038 [pool-3-thread-1] WARN i.a.c.HeartbeatMonitor$HeartbeatTask(handleException):112 - Cancelling job, workload is in a terminal state
io.airbyte.api.client.ApiException: Gone
at io.airbyte.workload.api.client.WorkloadApiClient.unit(WorkloadApiClient.kt:417)
at io.airbyte.workload.api.client.WorkloadApiClient.workloadRunning(WorkloadApiClient.kt:58)
at io.airbyte.connectorSidecar.HeartbeatMonitor$HeartbeatTask.run(HeartbeatMonitor.kt:93)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
2026-08-24 07:07:36,690 [main] INFO i.a.c.ConnectorWatcher(markWorkloadSuccess):207 - Marking workload ec27084b-13f2-4883-acc5-57e11896249f_84_0_check as successful
2026-08-24 07:07:36,801 [main] ERROR i.a.c.ConnectorWatcher(handleException):216 - Error performing operation: io.airbyte.api.client.ApiException
io.airbyte.api.client.ApiException: Gone
at io.airbyte.workload.api.client.WorkloadApiClient.unit(WorkloadApiClient.kt:417)
at io.airbyte.workload.api.client.WorkloadApiClient.workloadSuccess(WorkloadApiClient.kt:56)
at io.airbyte.connectorSidecar.ConnectorWatcher.markWorkloadSuccess(ConnectorWatcher.kt:208)
at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:95)
at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:22)
at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt)
2026-08-24 07:07:37,254 [main] INFO i.a.c.ConnectorWatcher(failWorkload):359 - Failing workload ec27084b-13f2-4883-acc5-57e11896249f_84_0_check.
2026-08-24 07:07:37,451 [main] INFO i.a.c.ConnectorWatcher(failWorkload):359 - Failing workload ec27084b-13f2-4883-acc5-57e11896249f_84_0_check.
2026-08-24 07:07:37,471 [main] INFO i.a.c.ConnectorWatcher(exitInternalError):271 - Deliberately exiting process with code 1.
for pod source-redshift-check-84-0-dzexm .kapa.ai
08/24/2026, 7:21 AMconnector-sidecar container finishes running a check operation and attempts to update the workload status (workloadRunning / workloadSuccess) via the Workload API, but receives an HTTP 410 Gone (ApiException: Gone) error [Github Issue (open): Proxy variables for sidecar container; Github Issue (open): sync pulling 0 records & exits out without raising error].
When the API returns Gone, it means the Airbyte platform has already marked the workload as terminated, cancelled, or cleaned up before the pod could report its success [Github Issue (open): Proxy variables for sidecar container; Github Issue (open): sync pulling 0 records & exits out without raising error]. Because the status update fails, the sidecar defaults to marking the workload as failed and exits with code 1 [Github Issue (closed): SFTP-JSON connector - Receive same sync error with any source connector ("State message does not contain id")].
Common Causes and Troubleshooting Steps 1. Workload Monitor Timed Out the Job Airbyte's background Workload Monitor monitors workload lifecycles. If a workload takes too long in a specific state or misses its heartbeat deadline, the monitor automatically terminates it [Workload Monitor].
⢠Action: Check if the check operation took a long time to start or complete (e.g., due to slow image pulls, cluster scheduling delays, or network latency).
⢠Inspect Kubernetes events and pod states for delays:kubectl describe pod source-redshift-check-84-0-dzexm -n your-namespace
kubectl get pods -n your-namespaceCheck for resource pressure (OOMKilled, CPU throttling, or pending scheduling) [Workload Monitor].
2. Temporal Workflow Persistence/Timeout Issues If you are running self-hosted Airbyte, persistence timeouts or connection bottlenecks in Temporal can cause workflow states to fall out of sync or time out, resulting in the workload record being removed before the sidecar reports back [Github Issue (open): sync pulling 0 records & exits out without raising error].
⢠Action:
⦠Check the logs of your Temporal server pods for context deadline exceeded or persistence.AppendHistoryTimeoutError [Github Issue (open): sync pulling 0 records & exits out without raising error].
⦠Verify the health of your metadata database (PostgreSQL) regarding CPU, connection limits, and I/O [Github Issue (open): sync pulling 0 records & exits out without raising error].
3. State Transition Conflicts / Workload API Configuration State transition race conditions or misalignments between the Workload API Server and Worker can cause status update failures [Github Issue (open): Jobs launching in k8s, but they are not removed when completed.; Github Issue (closed): [source-postgres] Connector setup fails with HTTP 502 error].
⢠Action: Ensure your workload-api-server, worker, and server pods are running the same version and are healthy [Github Issue (closed): [source-postgres] Connector setup fails with HTTP 502 error]. If running multiple replicas of workload-api-server, verify if scaling down temporarily resolves race conditions [Github Issue (open): Jobs launching in k8s, but they are not removed when completed.]. š 0 button š 0 button ā¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.Dilyan Stoev
08/24/2026, 1:12 PMkapa.ai
08/24/2026, 1:12 PM0 (e.g., replication-job-186-attempt-0) [Documentation Platform > Understanding-airbyte > Workloads & jobs>; Documentation Platform > Understanding-airbyte > Airbyte Databases Data Catalog>].
2. If attempt 0 encounters a failure, the Airbyte platform initiates retries within the same job execution, incrementing the attempt number (e.g., attempt-1, attempt-2) [Documentation Platform > Understanding-airbyte > Workloads & jobs>; Github Issue (open): Jobs launching in k8s, but they are not removed when completed.].
Therefore, having a pod named replication-job-186-attempt-1 indicates that replication-job-186-attempt-0 was previously created for that job run [Documentation Platform > Understanding-airbyte > Workloads & jobs>; Github Issue (open): Jobs launching in k8s, but they are not removed when completed.]. š 0 button š 0 button ā¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.Dilyan Stoev
08/24/2026, 1:13 PMkapa.ai
08/24/2026, 1:13 PMreplication-job-<job_id>-attempt-<attempt_number>.log (for example, replication-job-186-attempt-0.log) or stored in a path pattern containing <job_id>/<attempt_number>/logs.log [Github Issue (open): [destination-s3] Sync fails on WRITE init: NPE in DestinationStreamFactory when generationId/syncId are null (platform 2.0.0, Helm 2.0.18, AWS cn-north-1); Forum Post: Error: Failure Origin: replication, Message: Something went wrong during replication].
⢠Finding the Exact Path via Database: In the Airbyte jobs database, the attempts table tracks each attempt and includes a log_path column indicating the exact path where logs for that attempt are stored, as well as a failure_summary column containing the JSON failure details [Jobs Database].
2. What to Search For Within the Logs Once you open the log file for Attempt 0 (replication-job-186-attempt-0.log), look toward the end of the attempt log (where the attempt terminates) for the following [Github Issue (open): Sync job marked succeeded with 0 records loaded when its only attempt dies early (no retry scheduled)]:
⢠Common Error Keywords: Search for ERROR, Exception, or Fail to locate specific error stack traces [Browsing logs].
⢠Failure Summary Blocks: Look for JSON log entries containing fields such as [Github Issue (open): Sync job marked succeeded with 0 records loaded when its only attempt dies early (no retry scheduled); Github Issue (closed): low code connector pull incorrect records than expected (much more)]:
⦠failureOrigin (source, destination, or replication)
⦠internalMessage or externalMessage
⦠stacktrace
⢠Pod Crash or Termination Codes: Check for non-zero exit codes (such as exit code 143 or killed processes indicating an OOM event or node eviction) [Github Issue (open): Sync job marked succeeded with 0 records loaded when its only attempt dies early (no retry scheduled); Forum Post: Error: Failure Origin: replication, Message: Something went wrong during replication].
3. Alternative: Check via the Airbyte UI You can also view or download the attempt logs directly from the UI without manually searching S3:
1. Navigate to the connection's Job History tab and select the sync job [Browsing logs].
2. Select View logs to inspect the specific attempt inline, or click Download logs to download the unioned logs across all attempts for that job [Browsing logs]. š 0 button š 0 button ā¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.Dilyan Stoev
08/24/2026, 1:22 PM2026-08-24 15:56:04 replication-orchestrator INFO Records read: 215000 (296 MB)
2026-08-24 15:57:12 replication-orchestrator INFO Records read: 220000 (303 MB)
2026-08-24 15:58:20 replication-orchestrator INFO Records read: 225000 (310 MB)
2026-08-24 15:59:02 replication-orchestrator ERROR DestinationReader error:
2026-08-24 15:59:02 replication-orchestrator INFO DestinationReader finished.
2026-08-24 15:59:02 replication-orchestrator ERROR DestinationWriter error:
2026-08-24 15:59:02 replication-orchestrator ERROR DestinationWriter error:
2026-08-24 15:59:02 replication-orchestrator INFO MessageProcessor finished.
2026-08-24 15:59:02 replication-orchestrator WARN Attempted to close a destination which is already closed.
2026-08-24 15:59:02 replication-orchestrator INFO DestinationWriter finished.
2026-08-24 15:59:02 replication-orchestrator ERROR SourceReader error:
2026-08-24 15:59:02 replication-orchestrator INFO SourceReader finished.
2026-08-24 15:59:02 replication-orchestrator ERROR runJobs failed; recording failure but continuing to finish.
2026-08-24 15:59:02 replication-orchestrator INFO Closing StateCheckSumCountEventHandler
2026-08-24 15:59:02 replication-orchestrator INFO Sync summary: {
"status" : "failed",kapa.ai
08/24/2026, 1:22 PMDestinationReader error: and DestinationWriter error: occur first at 15:59:02 [Github Issue (open): S3 Datalake Connector: Failures marked as success. Main branch never created.; Github Issue (open): Firestore connector not letting me sync properly].
2. The orchestrator attempts to close the destination and message queues (WARN Attempted to close a destination which is already closed.) [Github Issue (open): S3 Datalake Connector: Failures marked as success. Main branch never created.; Github Issue (open): Firestore connector not letting me sync properly].
3. The SourceReader error: occurs immediately after as a downstream consequence (often due to a broken pipe when the destination process terminates unexpectedly) [Github Issue (open): S3 Datalake Connector: Failures marked as success. Main branch never created.; Github Issue (open): SourceReader can hang after destination failure due to blocking reads ignoring cancellation].
How to Find the Exact Root Cause The snippet cuts off right at "status" : "failed". To see the exact exception and root cause, look at the lines immediately following your snippet in the log file:
1. Check the Failures: block:
Right after the Sync summary: { ... } block, Airbyte logs a Failures: [ ... ] JSON object [Github Issue (open): S3 Datalake Connector: Failures marked as success. Main branch never created.; Github Issue (closed): replication-orchestrator ERROR]. Look for:
⦠"failureOrigin": "destination"
⦠"internalMessage"
⦠"stacktrace"
1. Search for Destination Stack Traces:
Search backwards from 15:59:02 in the log file for words like Exception, Caused by:, SQLSyntaxErrorException, timeout, or non-zero exit codes (e.g., Destination process exited with non-zero exit code) to find the connector's actual error message [Browsing logs; Github Issue (closed): replication-orchestrator ERROR]. š 0 button š 0 button ā¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.Dilyan Stoev
08/24/2026, 1:26 PM2026-08-24 15:59:02 replication-orchestrator INFO Failures: [ {
"failureOrigin" : "destination",
"internalMessage" : "Destination process exited with non-zero exit code 143",
"externalMessage" : "Something went wrong within the destination connector",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 186,
"connector_command" : "write"
},
"stacktrace" : "io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 143\n\tat io.airbyte.container.orchestrator.worker.DestinationReader.run(ReplicationTask.kt:55)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invokeSuspend(ReplicationWorker.kt:152)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invoke(ReplicationWorker.kt)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invoke(ReplicationWorker.kt)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invokeSuspend(AsyncUtils.kt:22)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invoke(AsyncUtils.kt)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invoke(AsyncUtils.kt)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:66)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)\n\tat kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165)\n\tat kotlinx.coroutines.BuildersKt.withContext(Unknown Source)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1.invokeSuspend(AsyncUtils.kt:21)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1787576342656
} ]kapa.ai
08/24/2026, 1:27 PMDestination process exited with non-zero exit code 143
What Exit Code 143 Means An exit code of 143 indicates that the destination process was killed by a SIGTERM signal (128 + 15) [Github Issue (open): Destination Process Exits with Code 143; Github Issue (open): Sync job marked succeeded].
Because SIGTERM comes from an external signal rather than the destination process terminating itself gracefully, this typically points to one of the following root causes:
1. Out of Memory (OOM) / Resource Constraints
Memory exhaustion on the destination container or Kubernetes node can cause the container/process to be terminated [Github Issue (open): GitHub Source Connector Consuming Excessive Memory; Github Issue (closed): OOM / Exit Code 143].
2. Kubernetes Pod Eviction or Node Shutdown
If the Kubernetes node was drained, preempted, shut down, or experienced pod eviction, Kubernetes sends a SIGTERM to the container [Github Issue (open): Destination Process Exits with Code 143; Github Issue (open): Sync job marked succeeded].
3. Liveness / Readiness Probe Failure or Orchestrator Termination
A failed Kubernetes health check or an orchestrator timeout can cause the container to receive a termination signal [Github Issue (open): Destination Process Exits with Code 143].
Recommended Next Steps to Investigate & Fix ⢠Check Kubernetes Events:
Inspect the events in your namespace around 15:59:02 to see if the pod was evicted or killed:kubectl get events --sort-by='.lastTimestamp' -n airbyte-namespaceCheck whether there are OOMKilled, Evicted, or NodeShutdown events [Github Issue (open): Destination Process Exits with Code 143; Github Issue (open): Sync job marked succeeded].
⢠Increase Memory Allocation:
If the failure happens consistently during large data syncs, consider increasing the memory requests/limits allocated to your worker/destination containers (e.g., via JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT) [Forum Post: Non-Zero Error Code 143; Forum Post: Customized source code 143].
⢠Inspect Destination Container Output:
Check if the destination container outputted any logs (e.g., OutOfMemoryError: Java heap space or network disconnects) immediately before exiting [Github Issue (open): Destination Process Exits with Code 143; Github Issue (open): GitHub Source Connector Consuming Excessive Memory]. š 0 button š 0 button ā¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.Dilyan Stoev
08/24/2026, 1:30 PM30m Normal Nominated pod/replication-job-186-attempt-0 Pod should schedule on: nodeclaim/production-nvi-karpenter-airbyte-worker-nodepool-j7qjh
29m Normal Evicted pod/replication-job-171-attempt-0 Evicted pod: Underutilized
29m Normal Killing pod/replication-job-171-attempt-0 Stopping container source
29m Normal Killing pod/replication-job-171-attempt-0 Stopping container destination
29m Normal Killing pod/replication-job-186-attempt-0 Stopping container destination
29m Normal Killing pod/replication-job-186-attempt-0 Stopping container source
29m Normal Evicted pod/replication-job-186-attempt-0 Evicted pod: Underutilized
29m Normal Killing pod/replication-job-186-attempt-0 Stopping container orchestratorkapa.ai
08/24/2026, 1:30 PMEvicted pod: Underutilized) to consolidate workloads or scale down nodes [Github Issue (closed): Sync attempt killed pod hangs].
2. SIGTERM / Exit Code 143:
When Kubernetes evicted the pod, the kubelet sent a SIGTERM to stop the containers (Stopping container destination, Stopping container source, Stopping container orchestrator) [Github Issue (open): Sync job marked succeeded].
3. Attempt 0 Failure & Retry:
The destination container terminated with exit code 143 (SIGTERM), causing replication-job-186-attempt-0 to fail and Airbyte to automatically launch replication-job-186-attempt-1 [Github Issue (open): Sync job marked succeeded].
How to Prevent This ⢠Prevent Node Eviction with Karpenter / Autoscaler Annotations:
Configure your job pods or autoscaler settings to prevent eviction during active runs (for example, by setting eviction protections such as karpenter.sh/do-not-disrupt: "true" or cluster-autoscaler.kubernetes.io/safe-to-evict: "false") [Github Issue (open): MySQL CDC sync jobs fail].
⢠Configure Karpenter Disruption Controls:
Adjust your Karpenter NodePool disruption settings (e.g., set consolidationPolicy: WhenEmpty instead of WhenUnderutilized, or configure disruption budgets) so nodes running active Airbyte sync jobs are not drained mid-flight [Github Issue (closed): Sync attempt killed pod hangs]. š 0 button š 0 button ā¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.