Hi @kapa.ai Failed to update workload status to launched. Workload may be reprocessed on restart.Exception: io.airbyte.api.client.ApiException: Gone Getting this error randomly in our pipelines ?

That

ApiException: Gone

when updating a workload to

LAUNCHED

is coming from the Workload API returning HTTP 410 (“Gone”) for that workload. From the logs in the proxy/sidecar issue, the same 410 is returned when the sidecar tries to heartbeat a workload that is already in a terminal state, and the server responds: >

Client error : 410 Gone {"message":"Heartbeat a workload in a terminal state"}

[[Proxy sidecar](https://github.com/airbytehq/airbyte/issues/56917)] While your exact message is during

updateStatusToLaunched

, the pattern is similar: the Workload API is rejecting changes/heartbeats for a workload whose lifecycle, from the API’s point of view, has already ended or is otherwise no longer valid. In that case, the launcher logs: > “Failed to update workload status to launched. Workload may be reprocessed on restart.” The knowledge sources don’t give a precise root cause beyond this (for example, whether it’s due to races, restarts, or DB cleanup), but they do indicate: - 4xx errors from the Workload API generally occur when the workload status is no longer in a mutable state (terminal / removed / invalid). [[Proxy sidecar](https://github.com/airbytehq/airbyte/issues/56917)] - When launcher status updates fail, the platform may retry or reprocess the workload on restart (as your log warns), which can lead to re-execution if not handled. There isn’t a documented, generic fix in the provided sources. They do not describe configuration changes or a patch to avoid the 410 “Gone” in this specific

updateStatusToLaunched

case, so I can’t safely recommend a concrete remediation beyond: - Be aware that these 410s indicate a mismatch between what the launcher thinks the workload state is and what the Workload API believes. - If this is happening frequently or causing duplicates/hangs, it likely warrants opening or updating a GitHub issue with full launcher + workload-api logs and your platform version, so the Airbyte team can investigate the lifecycle edge case. If you can share your Airbyte version and any surrounding logs (especially from workload-launcher and the workload-api service), that would help correlate with the behaviors seen in the linked issues.

@kapa.ai this is what i see in temporal logs {"level":"error","ts":"2025-12-18T000306.058Z","msg":"Update workflow execution operation failed.","shard-id":4,"address":"0.0.0.0:7234","wf-namespace-id":"9acea73b-9ef8-471b-9ca5-77f790fe6082","wf-id":"replication_62561","wf-run-id":"019b2ec3-5643-7446-9a73-3532e0ec5615","store-operation":"update-wf-execution","error":"context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:510","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:154\ngo.temporal.io/server/service/history/workflow.updateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:510\ngo.temporal.io/server/service/history/workflow.(*TransactionImpl).UpdateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:188\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionWithNew\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/context.go:654\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionAsActive\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/context.go:502\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).updateWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:852\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).executeUserTimerTimeoutTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:208\ngo.temporal.io/server/service/history.(*timerQueueActiveTaskExecutor).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/timer_queue_active_task_executor.go:123\ngo.temporal.io/server/service/history/queues.(*activeStandbyExecutor).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/queues/active_standby_executor.go:67\ngo.temporal.io/server/service/history/queues.(*executableImpl).Execute\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/queues/executable.go:374\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:223\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:62\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:89\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:63\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/tasks/fifo_scheduler.go:211"} {"level":"warn","ts":"2025-12-18T000306.059Z","msg":"Fail to process task","shard-id":4,"address":"0.0.0.0:7234","component":"timer-queue-processor","wf-namespace-id":"9acea73b-9ef8-471b-9ca5-77f790fe6082","wf-id":"replication_62561","wf-run-id":"019b2ec3-5643-7446-9a73-3532e0ec5615","queue-task-key":{"FireTime":"2025-12-18T000252.433Z","TaskID":1219494862},"queue-task-type":"UserTimer","queue-task":"9acea73b-9ef8-471b-9ca5-77f790fe6082/replication_62561/019b2ec3-5643-7446-9a73-3532e0ec5615","wf-history-event-id":17,"error":"context deadline exceeded","error-type":"persistence.AppendHistoryTimeoutError","unexpected-error-attempts":1,"lifecycle":"ProcessingFailed","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/log/lazy_logger.go:63"}

A message from kapa.ai