@kapa.ai after upgrading to 1.8.5 it looks like a lot of syncs are stuck in 'starting' and the pods are not even there

This is a known issue after upgrading Airbyte, and the most common root cause is a dataplane group mismatch — workloads are being routed to one

dataplane_group

, but the workload-launcher is listening to a different one, so pods are never launched. Diagnosis Run this query against your Airbyte database to check for a mismatch:

SELECT w.id AS workspace_id,
       w.dataplane_group_id AS workspace_group,
       d.id AS dataplane_id,
       d.dataplane_group_id AS dataplane_group,
       d.enabled
FROM workspace w
JOIN dataplane d ON d.enabled = true
WHERE w.dataplane_group_id <> d.dataplane_group_id;

The result should return zero rows. If it doesn't, you have a mismatch that needs to be corrected. [Connector-Sidecar Issue] Also check for duplicate dataplane groups:

SELECT * FROM public.dataplane_group;

If you see two entries (e.g. one from before the upgrade and one created after), jobs may be routed to the old group while the launcher listens to the new one. [Syncs Stuck Issue] Fix Options Option 1 — Update the workspace to point to the correct dataplane group:

-- Find the correct group ID first
SELECT * FROM public.dataplane_group;

-- Then update the workspace
UPDATE workspace
SET dataplane_group_id = '<correct-dataplane-group-id>'
WHERE dataplane_group_id = '<wrong-dataplane-group-id>';

Then restart the workload-launcher:

kubectl rollout restart deployment/airbyte-workload-launcher -n <namespace>

[Syncs Stuck Issue] Option 2 — Use the internal API (no direct DB access needed):

# Get the correct group from the dataplanes API
CORRECT_GROUP=$(kubectl exec -n <NS> deploy/airbyte-server -- \
  curl -s http://localhost:8001/api/public/v1/dataplanes \
  | python3 -c "import sys,json; print(json.load(sys.stdin)[0]['regionId'])")

# Update the workspace to match
kubectl exec -n <NS> deploy/airbyte-server -- \
  curl -s -X POST http://localhost:8001/api/v1/workspaces/update \
  -H "Content-Type: application/json" \
  -d "{\"workspaceId\":\"<WORKSPACE_ID>\",\"dataplaneGroupId\":\"$CORRECT_GROUP\"}"

[Dataplane Group Mismatch Issue] Option 3 — Prevent the issue on future upgrades: Add this to your Helm values before upgrading:

global:
  dataplaneGroups:
    defaultDataplaneGroupName: "AUTO"

This ensures the default dataplane group name is consistent across upgrades. [Syncs Stuck Issue] Confirming the Fix After applying the fix, check the workload-launcher logs for lines like:

INFO  Resuming ApiQueuePoller with <group-id> and high
INFO  Resuming ApiQueuePoller with <group-id> and default

If you see these, the launcher is now polling the correct group and pods should start launching. [Dataplane Group Mismatch Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I can see a lot of these logs

2026-06-05 05:37:45,996 [parallel-1] WARN i.a.w.l.p.c.WorkloadApiQueuePoller(handlePollError):112 - Error encountered in poller: io.airbyte.api.client.ApiException: Unauthorized

in the workload launcher

The

Unauthorized

error in the workload-launcher poller points to a token/authentication mismatch between the workload-launcher and the workload API server, rather than (or in addition to) the dataplane group mismatch discussed earlier. Fix: Align the Bearer Tokens Generate a new shared token and apply it consistently across the relevant deployments:

# Generate a token
TOKEN=$(openssl rand -hex 24)
echo "Generated token: $TOKEN"

# Apply to workload-api-server, worker, and server
kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \
  WORKLOAD_API_AUTH_HEADER=Authorization \
  WORKLOAD_API_BEARER_TOKEN=$TOKEN

kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \
  WORKLOAD_API_BEARER_TOKEN="$TOKEN" \
  INTERNAL_API_AUTH_TOKEN="$TOKEN"

kubectl -n airbyte-abctl set env deployment/airbyte-abctl-server \
  WORKLOAD_API_AUTH_HEADER=Authorization \
  WORKLOAD_API_BEARER_TOKEN=$TOKEN

# Restart the affected deployments
kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-workload-api-server
kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-worker
kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-server

# Wait for rollouts
kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-workload-api-server
kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-worker

> Note: Adjust the namespace (

-n airbyte-abctl

) to match your deployment if different. [Connector Setup 502 Issue] Additional Check Also verify that

API_AUTHORIZATION_ENABLED=true

is set in your

airbyte-airbyte-env

ConfigMap — without this, the workload output endpoint may return 404 or reject auth headers entirely. [Connector-Sidecar Auth Issue] Important: The knowledge sources don't contain a definitive explanation of why the tokens become misaligned after a 1.8.5 upgrade specifically, so if the above doesn't resolve it, I'd recommend checking the Airbyte Community Slack or opening a GitHub issue with your specific logs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai this is what I see in the worker pod:

Failure in poller thread Workflow Poller taskQueue="CHECK_CONNECTION", namespace="default": 2
io.grpc.StatusRuntimeException: INVALID_ARGUMENT: task queue name cannot start with reserved prefix /_sys/
 at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:351)
 at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:332)
 at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174)
 at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollWorkflowTaskQueue(WorkflowServiceGrpc.java:5634)
 at io.temporal.internal.worker.WorkflowPollTask.doPoll(WorkflowPollTask.java:174)
 at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:155)
 at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:49)
 at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:336)
 at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:296)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 at java.base/java.lang.Thread.run(Thread.java:1583)

The knowledge sources do not contain enough information to provide a confident, specific answer about this exact error. That said, based on what the error message itself tells us:

INVALID_ARGUMENT: task queue name cannot start with reserved prefix /_sys/

The worker is attempting to poll a Temporal task queue whose name starts with

/_sys/

, which is a reserved internal Temporal prefix. This is visible in the Temporal logs you shared earlier, where queues like

/_sys/CHECK_CONNECTION/2

appear. The worker should be polling

CHECK_CONNECTION

, not

/_sys/CHECK_CONNECTION/...

. What This Suggests This looks like a misconfiguration of the task queue name being passed to the Temporal worker — the worker is receiving or constructing a queue name that includes the internal

/_sys/

prefix, which Temporal does not allow external workers to poll directly. [Temporal Context Deadline Issue] Recommended Steps 1. Check your Temporal-related environment variables in the worker deployment for any misconfigured queue name overrides:kubectl -n namespace get deployment airbyte-worker -o yaml | grep -i temporal 2. Check your Airbyte version compatibility — this combination of symptoms (Unauthorized in workload-launcher + malformed task queue names in worker) after a 1.8.5 upgrade may indicate a partially failed upgrade where some components are running mismatched versions:kubectl -n namespace get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}' 3. Consider a full pod restart of all Airbyte components to ensure all pods are running the same version:kubectl -n namespace rollout restart deployment I'd strongly recommend opening a GitHub issue or checking the Airbyte Community Slack with these specific logs, as this combination of errors after a 1.8.5 upgrade is not covered in detail in the available knowledge sources. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai the images have all the 1.8.5 tag except for the

aribyte-temporal

deployment where the image is

temporalio/auto-setup:1.27.2

This is actually a known open issue with the Airbyte Helm chart. The

temporalio/auto-setup:1.27.2

image is the expected/pinned image for the Temporal deployment in current Airbyte Helm chart versions — it is not a version mismatch causing your problems. [Deprecated auto-setup Issue] However, it is worth noting that

temporalio/auto-setup

is officially deprecated as of Temporal v1.30.1. The recommended migration path is: • Switch to

temporalio/server

image with

AUTO_SETUP=true

environment variable This replacement has been escalated to the Airbyte internal team but has not yet been fixed in the Helm chart as of the time of writing. [Deprecated auto-setup Issue] What This Means For Your Current Issue The

/_sys/

task queue error you're seeing in the worker is unlikely to be caused by the Temporal image version. Your deployment looks correct — all Airbyte components on

1.8.5

with

temporalio/auto-setup:1.27.2

is the expected state. Given the combination of errors you're seeing: 1.

Unauthorized

in the workload-launcher poller 2.

INVALID_ARGUMENT: task queue name cannot start with reserved prefix /_sys/

in the worker The most likely remaining culprits are: • The auth token mismatch between workload-launcher and workload API server (as discussed previously) • Potentially stale Temporal workflows from a previous version carrying over — these can cause

NonDeterministicException

errors and malformed queue polling [Dataplane Group Mismatch Issue] I'd recommend resolving the auth token issue first, then checking if stale Temporal workflows need to be terminated via

tctl

. The knowledge sources don't contain enough detail to say definitively what's causing the

/_sys/

prefix error specifically. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I restarted the dpeloyment and now the workload-launcher pod crashes with:

2026-06-05 06:37:32,743 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):56 - Starting pod sweeper cycle in namespace [airbyte]...
2026-06-05 06:37:32,748 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):67 - Will sweep Succeeded pods older than 2026-06-05T06:27:32.744Z (UTC).
2026-06-05 06:37:32,748 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):70 - Will sweep unsuccessful pods older than 2026-06-05T04:37:32.744Z (UTC).
2026-06-05 06:37:33,937 [main] INFO i.a.f.ConfigFileClient(<init>):150 - path /flags does not exist, will return default flag values
2026-06-05 06:37:34,155 [scheduled-executor-thread-1] INFO i.a.w.l.p.c.WorkloadApiQueueConsumer(initialize):39 - Initializing ApiQueueConsumer for a77e3023-e837-46ac-bd5b-ec30a62a9ee5
2026-06-05 06:37:34,219 [scheduled-executor-thread-1] INFO i.a.w.l.p.c.WorkloadApiQueuePoller(initialize):53 - Initalizing ApiQueuePoller with a77e3023-e837-46ac-bd5b-ec30a62a9ee5 and default
2026-06-05 06:37:34,240 [main] INFO i.a.w.l.a.DataplaneIdentityService(initialize):53 - Running as 266bd192-c3a1-497a-9628-45282b5568c7 (AUTO) for AUTO
2026-06-05 06:37:34,250 [main] INFO i.m.r.Micronaut(start):101 - Startup completed in 5659ms. Server Running: <http://airbyte-workload-launcher-7645999b88-qxfcq:8016>
2026-06-05 06:37:34,305 [scheduled-executor-thread-1] INFO i.a.w.l.p.c.WorkloadApiQueuePoller(initialize):53 - Initalizing ApiQueuePoller with a77e3023-e837-46ac-bd5b-ec30a62a9ee5 and high
2026-06-05 06:37:34,306 [Thread-2] INFO i.a.w.l.ClaimedProcessor(getWorkloadList):91 - requesting workload list: WorkloadListRequest(dataplane=[266bd192-c3a1-497a-9628-45282b5568c7], status=[claimed], updatedBefore=null)
2026-06-05 06:37:34,966 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):134 - Completed pod sweeper cycle.
2026-06-05 06:37:35,080 [Thread-2] ERROR i.a.w.l.StartupApplicationEventListener(onApplicationEvent$lambda$1):43 - Failed to retrieve and resume claimed workloads, exiting.
io.airbyte.api.client.ApiException: Unauthorized
 at io.airbyte.workload.api.client.WorkloadApiClient.workloadList(WorkloadApiClient.kt:283)
 at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList$lambda$9(ClaimedProcessor.kt:112)
 at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
 at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
 at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
 at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
 at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
 at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
 at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList(ClaimedProcessor.kt:110)
 at io.airbyte.workload.launcher.ClaimedProcessor.retrieveAndProcess(ClaimedProcessor.kt:57)
 at io.airbyte.workload.launcher.StartupApplicationEventListener.onApplicationEvent$lambda$1(StartupApplicationEventListener.kt:39)
 at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

The crash is very clear — the workload-launcher is hitting

Unauthorized

on startup when trying to call

workloadList

, and it exits immediately because of this. This confirms the auth token mismatch is the root cause of all your issues. Looking at the logs, there is one positive sign:

Running as 266bd192-c3a1-497a-9628-45282b5568c7 (AUTO) for AUTO

The dataplane group is now correctly identified as

AUTO

, so that part is resolved. The only remaining blocker is the authentication between the workload-launcher and the workload-api-server. Steps to Fix 1. Check what token the workload-launcher is currently using:

kubectl -n <namespace> get deployment airbyte-workload-launcher -o yaml | grep -i token

2. Check what token the workload-api-server expects:

kubectl -n <namespace> get deployment airbyte-workload-api-server -o yaml | grep -i token

3. Check the shared ConfigMap for the token:

kubectl -n <namespace> get configmap airbyte-airbyte-env -o yaml | grep -i token

4. If they don't match, patch them to use the same token:

TOKEN=$(openssl rand -hex 24)

kubectl -n <namespace> set env deployment/airbyte-workload-launcher \
  WORKLOAD_API_BEARER_TOKEN=$TOKEN

kubectl -n <namespace> set env deployment/airbyte-workload-api-server \
  WORKLOAD_API_BEARER_TOKEN=$TOKEN

kubectl -n <namespace> rollout restart deployment/airbyte-workload-launcher
kubectl -n <namespace> rollout restart deployment/airbyte-workload-api-server

5. Confirm the launcher starts successfully by watching for these lines in the logs:

Resuming ApiQueuePoller with <group-id> and high
Resuming ApiQueuePoller with <group-id> and default

The absence of these lines (only seeing

PodSweeper

cycles) means the launcher is still not authenticating correctly. [Syncs Stuck Issue; Dataplane Group Mismatch Issue] > Note: The knowledge sources do not contain the exact mechanism by which the token becomes misaligned after a 1.8.5 upgrade. If the above steps don't resolve it, I'd recommend checking your Helm

values.yaml

to ensure

WORKLOAD_API_BEARER_TOKEN

is explicitly set and consistent across all relevant deployments, rather than relying on auto-generated values that may differ between rollouts. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai checking that I saw:

- name: WORKLOAD_API_BEARER_TOKEN                                                                                                                                                                                                        valueFrom:                                                                                                                                                                                                                               secretKeyRef:                                                                                                                                                                                                                            key: WORKLOAD_API_BEARER_TOKEN                                                                                                                                                                                                         name: airbyte-airbyte-secrets                                                                                                                                                                                                    - name: WORKLOAD_API_BEARER_TOKEN_SECRET_NAME                                                                                                                                                                                            value: airbyte-airbyte-secrets                                                                                                                                                                                                       - name: WORKLOAD_API_BEARER_TOKEN_SECRET_KEY                                                                                                                                                                                             value: WORKLOAD_API_BEARER_TOKEN

If I check in the cluster, I cannot see the

airbyte-airbyte-secrets

but that's actually there in the dev cluster! I deploy with the same helm chart, how is that even possible that the secrets are not there

The missing

airbyte-airbyte-secrets

secret is almost certainly the root cause of everything — the workload-launcher can't authenticate because the secret it's trying to read the token from simply doesn't exist in this cluster. Why Does It Exist in Dev But Not Here? The

airbyte-airbyte-secrets

secret is created as a Helm pre-install hook (

helm.sh/hook: pre-install,pre-upgrade

). There are a few known reasons it can go missing: [Stuck on EKS+Helm] 1. The pre-install hook failed silently during the 1.8.5 upgrade, leaving the secret uncreated 2. The secret was manually deleted at some point and the hook didn't re-run (hooks only run on install/upgrade, not on pod restarts) 3. A previous failed upgrade left the cluster in a partial state Verify It's Missing

kubectl -n <namespace> get secret airbyte-airbyte-secrets
# Should return "Error from server (NotFound)"

# Also check what secrets ARE present
kubectl -n <namespace> get secrets | grep airbyte

Fix: Recreate the Secret Manually

TOKEN=$(openssl rand -hex 24)

kubectl -n <namespace> create secret generic airbyte-airbyte-secrets \
  --from-literal=WORKLOAD_API_BEARER_TOKEN=$TOKEN

Then restart the affected deployments:

kubectl -n <namespace> rollout restart deployment/airbyte-workload-launcher
kubectl -n <namespace> rollout restart deployment/airbyte-workload-api-server
kubectl -n <namespace> rollout restart deployment/airbyte-worker

Prevent It Happening Again To avoid the hook failing silently on future upgrades, trigger a proper Helm upgrade so the pre-install/pre-upgrade hook recreates the secret correctly:

helm upgrade airbyte airbyte/airbyte \
  --namespace <namespace> \
  --values values.yaml \
  --version 1.8.5

> Important: Make sure the

TOKEN

value you use when manually creating the secret matches what gets set in

airbyte-airbyte-secrets

after the upgrade — otherwise you'll need to patch it again. The safest approach is to let the Helm upgrade recreate it, then check the value and align all deployments to it. [Preconfiguring Secrets] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.