@kapa.ai I have this issue on my k8s install of airbyte

Stage Pipeline Exception: io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: Failed to delete pods for mutex key: 848f70f4-4a49-485e-9825-15d8faf14522.
message: io.airbyte.workers.exception.KubeClientException: Failed to delete pods for mutex key: 848f70f4-4a49-485e-9825-15d8faf14522.

The knowledge sources confirm that this error comes from the MUTEX stage of the workload-launcher, when Airbyte tries to delete any existing Kubernetes pods for a given

mutex key

before launching a new sync. [S3 Datalake; workload liveness] In normal logs you’d see something like:

APPLY Stage: MUTEX — (workloadId=..._sync)
Mutex key: ... specified for workload: ... Attempting to delete existing pods...
Existing pods for mutex key: ... deleted.
APPLY Stage: ARCHITECTURE — ...
APPLY Stage: LAUNCH — ...

In your case, the

KubeClientException: Failed to delete pods for mutex key

means that Kubernetes pod deletion failed, so the MUTEX stage throws and the pipeline aborts before

ARCHITECTURE

/

LAUNCH

. The shared issues don’t include a direct fix for this specific error, but they show how the team is debugging similar KubeClient / workload-launcher problems and what they recommend checking. [Replication pod fail] Based on those, here are the concrete next steps you can take: 1. Check Kubernetes events for pod deletion failures Look for

FailedDelete

or related reasons:

kubectl get events -n <your-airbyte-namespace> --field-selector reason=FailedDelete

Also list events for the specific pods that should be deleted (they will usually be `replication-job-

<jobId>

-attempt-`n):

kubectl get events -n <your-airbyte-namespace> | grep replication-job

2. Inspect the pods that match the mutex key The mutex key corresponds to a connection/job; there may be stuck pods that Airbyte can’t delete:

kubectl get pods -n <your-airbyte-namespace> | grep replication-job

For any pods that look stale or stuck in

Terminating

, try deleting one manually and see if Kubernetes reports an error:

kubectl delete pod <pod-name> -n <your-airbyte-namespace>

If manual deletion also fails, the problem is at the Kubernetes level (finalizers, volume issues, node problems, etc.). 3. Check workload-launcher logs around the error The issues recommend looking at the workload-launcher logs for more detail on KubeClientExceptions: [Replication pod fail]

kubectl logs -l app.kubernetes.io/name=workload-launcher \
     -n <your-airbyte-namespace> --tail=200

Search for your mutex key

848f70f4-4a49-485e-9825-15d8faf14522

or for

Failed to delete pods for mutex key

. 4. Check for resource / cluster constraints Some related issues are caused by Kubernetes resource or API problems rather than Airbyte logic. The maintainers suggest checking node resources: [Replication pod fail]

kubectl describe nodes | grep -A 5 "Allocated resources"

If the API server or nodes are under heavy pressure, pod lifecycle operations (including delete) can fail intermittently. 5. If the error persists, collect data for a bug report Since there are already open issues around KubeClientExceptions and workload lifecycle on k8s, you may be hitting a related platform bug. [Replication pod fail; Jobs not removed] When opening/adding to a GitHub issue, include: ◦ Airbyte version and Helm chart version ◦ Number of replicas of

workload-api-server

/

workload-launcher

◦ The exact error stack trace ◦ Output of: ▪︎

kubectl get pods -n

your-airbyte-namespace ▪︎

kubectl get events -n

your-airbyte-namespace

--field-selector reason=FailedDelete

▪︎ Relevant

workload-launcher

logs (with secrets redacted) The knowledge sources don’t provide a specific configuration change or patch that definitively fixes this exact

Failed to delete pods for mutex key

error, so the best you can do now is use the steps above to pinpoint whether it’s a Kubernetes-level deletion problem or a reproducible Airbyte platform bug, and then share that detail in an issue for further help. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai The pod airbyte is looking for has been deleted, it should be looking for it

New message text here

@kapa.ai it is from a sync that has been running and looping on this error for 10hours, how do i restore it to a state that doesn't have this error ?

The knowledge sources don’t describe this exact looping failure mode or a one-click “reset” for a stuck sync, so I can’t give a guaranteed fix. I can, however, outline the safest way to get back to a clean state, based on how Airbyte behaves on Kubernetes in similar cases. From the issues, when things work normally you see: ~ text Mutex key: ... Attempting to delete existing pods... Mutex key: ... found no existing pods. Continuing... APPLY Stage: ARCHITECTURE — APPLY Stage: LAUNCH —

[[workload liveness](https://github.com/airbytehq/airbyte/issues/57011); [sync pulling 0 records](https://github.com/airbytehq/airbyte/issues/70984)]

In your case, the MUTEX stage keeps failing with `KubeClientException` and the job is effectively stuck. To restore to a non-broken state, you want to:

### 1. Stop the stuck sync in Airbyte

The forum thread about "all syncs hanging for days" indicates that cancelling the hanging syncs in the UI is what finally cleared them. [[All syncs hanging](https://discuss.airbyte.io/t/1916)]

- In the Airbyte UI, open the connection that's stuck.
- Click **Cancel** on the running job.
- Wait until the job status changes from "Running" to "Cancelled" (or "Failed").

If the UI won't let you cancel, you may need to restart the platform components (see step 3).

### 2. Clean up any leftover replication pods in Kubernetes

Even if you believe the pod is gone, double‑check and remove any stale ones:

kubectl get pods -n your-airbyte-namespace | grep replication-job ~ For any pods that look related to that connection/job and are

Running

or

Terminating

, delete them:

kubectl delete pod <pod-name> -n <your-airbyte-namespace>

This is similar to what users do when pods accumulate or get stuck, before relying on the pod-sweeper. [Destination pods terminating; Jobs not removed] 3. (If needed) Restart workload components If the job keeps re‑entering the same error loop even after cancellation and pod cleanup, restart the workload components so the launcher’s internal state is reset (the docs and issues don’t give exact commands, but users commonly restart pods when jobs hang). [All syncs hanging] For example:

kubectl rollout restart deployment -n <your-airbyte-namespace> \
  -l app.kubernetes.io/name=workload-launcher

kubectl rollout restart deployment -n <your-airbyte-namespace> \
  -l app.kubernetes.io/name=workload-api-server

(Adjust labels/names to your install.) 4. Re‑run the sync Once: • The old job is Cancelled/Failed in the UI, • No replication pods for that job remain, • And (if needed) workload components have been restarted, start a new sync for that connection from the UI. It should go through the MUTEX stage again and, if no pods exist, log “found no existing pods. Continuing…” and proceed. [workload liveness] ------------------------- Because the sources don’t document this exact bug, if you can reproduce it after these steps, I’d strongly recommend opening or updating a GitHub issue with: • Full logs around the MUTEX stage (with the mutex key), • Confirmation that no pods exist when the error occurs, • Airbyte version and k8s deployment details, similar to what’s requested in the existing investigations. [Replication pod fail; Jobs not removed] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.