<@U04S8PLJWMT> having problems with airbyte-worklo...
# ask-ai
e
@kapa.ai having problems with airbyte-workload-launcher. auth is disabled 2025-12-02 124351,538 [Thread-2] ERROR i.a.w.l.StartupApplicationEventListener(onApplicationEvent$lambda$1):43 - Failed to retrieve and resume claimed workloads, exiting. io.airbyte.api.client.ApiException: Unauthorized at io.airbyte.workload.api.client.WorkloadApiClient.workloadList(WorkloadApiClient.kt:288) at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList$lambda$9(ClaimedProcessor.kt:112) at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243) at dev.failsafe.Functions.lambda$get$0(Functions.java:46) at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112) at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList(ClaimedProcessor.kt:110) at io.airbyte.workload.launcher.ClaimedProcessor.retrieveAndProcess(ClaimedProcessor.kt:57) at io.airbyte.workload.launcher.StartupApplicationEventListener.onApplicationEvent$lambda$1(StartupApplicationEventListener.kt:39) at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)
k
The
io.airbyte.api.client.ApiException: Unauthorized
from
airbyte-workload-launcher
means its requests to the Workload API are being rejected with 401, even if you believe “auth is disabled”. From similar issues, this almost always comes down to the Workload API bearer token / header not matching between the launcher and the Workload API server (and sometimes worker). [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502 comment](https://github.com/airbytehq/airbyte/issues/65567)] Key points from the reports: - In local
abctl
installs, the fix was to recreate the secret that holds the bearer token used by the Workload API and worker: [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574)]
Copy code
bash   KUBECONFIG="${HOME}/.airbyte/abctl/abctl.kubeconfig"   NAMESPACE="airbyte-abctl"   SECRET_NAME="airbyte-abctl-workload-api-secrets"    WORKLOAD_TOKEN=$(openssl rand -hex 32)    kubectl delete secret "$SECRET_NAME" -n "$NAMESPACE" --ignore-not-found --kubeconfig "$KUBECONFIG"    kubectl create secret generic "$SECRET_NAME" \     --from-literal=bearer-token="$WORKLOAD_TOKEN" \     -n "$NAMESPACE" \     --kubeconfig "$KUBECONFIG"
- In another case, all Workload-related deployments had to share the same
WORKLOAD_API_AUTH_HEADER
and `WORKLOAD_API_BEARER_TOKEN`: [[postgres 502 comment](https://github.com/airbytehq/airbyte/issues/65567)]
Copy code
bash   TOKEN=$(openssl rand -hex 24)    kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \     WORKLOAD_API_AUTH_HEADER=Authorization \     WORKLOAD_API_BEARER_TOKEN=$TOKEN    kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \     WORKLOAD_API_AUTH_HEADER=Authorization \     WORKLOAD_API_BEARER_TOKEN=$TOKEN    kubectl -n airbyte-abctl set env deployment/airbyte-workload-launcher \     WORKLOAD_API_AUTH_HEADER=Authorization \     WORKLOAD_API_BEARER_TOKEN=$TOKEN
Given your error is in
workloadList
from the launcher, the concrete steps I’d recommend: 1. Identify the namespace and deployment names you’re using (Helm vs
abctl
may differ). 2. Inspect env vars on: -
airbyte-workload-api-server
-
airbyte-workload-launcher
- (and
worker
if present) and confirm
WORKLOAD_API_AUTH_HEADER
and
WORKLOAD_API_BEARER_TOKEN
are identical. 3. If in doubt, generate a new token and set it on all of them as in the snippet above, then restart those pods. If you truly have a mode where auth is supposed to be disabled entirely, the knowledge sources do not document how to configure that or how the launcher behaves in that case, so I can’t reliably describe additional flags or settings beyond ensuring the tokens line up.
e
@kapa.ai I'm on helm 1.9.1
k
A message from kapa.ai
For Helm-based installs in the 1.8–1.9 range, the
Unauthorized
errors between the workload launcher and the Workload API are the same class of issue that people hit with
abctl
in 1.8.x: the Workload API auth header/token are either missing or not consistent between components. However, all concrete remediation steps in the knowledge sources are for
abctl
/ local
deployments (
airbyte-abctl-*
deployments), not the Helm chart you’re using. [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502](https://github.com/airbytehq/airbyte/issues/65567); [1.8.3 502](https://github.com/airbytehq/airbyte/issues/66692)] Those sources show: - The failure mode:
io.airbyte.api.client.ApiException: Unauthorized
from workload-related components when calling the Workload API. [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] - The workaround: manually setting the same
WORKLOAD_API_BEARER_TOKEN
(and sometimes
WORKLOAD_API_AUTH_HEADER
) environment variables on the workload API server and worker, then restarting those deployments. [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] What’s not present in the knowledge sources: - Any Helm 1.9.1–specific configuration knobs (values.yaml keys) for disabling workload auth or wiring those tokens for Helm deployments. - Any official doc or issue explicitly describing how to fix this
Unauthorized
error on Helm 1.9.1. Because of that, I cannot provide a Helm-1.9.1-specific, chart-values-based fix from these sources. What you can take from the existing information: - The root cause is almost certainly that the Helm deployment of: -
airbyte-workload-api-server
-
airbyte-workload-launcher
(and possibly
airbyte-worker
) does not have aligned
WORKLOAD_API_BEARER_TOKEN
/
WORKLOAD_API_AUTH_HEADER
, similar to the
abctl
cases. [[postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] - A likely workaround (conceptually) would be to ensure those env vars are set to the same values across those deployments and then restart, just as users did with
abctl
. [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] But since there is no documentation here on how Helm 1.9.1 exposes or manages those variables, I can’t reliably tell you: - Which exact Helm values to set, or - Whether there is a Helm-native way to “disable” this auth for 1.9.1. So, based strictly on the available sources: - The problem you’re seeing on Helm 1.9.1 is consistent with known Workload API auth/token mismatches. - The only documented fixes are for `abctl`/local and involve manually aligning
WORKLOAD_API_BEARER_TOKEN
(and optionally the header) between the Workload API and worker/launcher. - The knowledge sources do not contain enough Helm-specific information for 1.9.1 to give an exact configuration snippet or official fix.
e
@kapa.ai still getting the error after setting:
Copy code
auth:
        enabled: false

      edition: community
k
A message from kapa.ai
auth.enabled: false
in the Helm values only disables user-facing auth; it does not disable the internal authentication between the workload components (workload-launcher, workload-api-server, worker). The
Unauthorized
you see is from the launcher calling the Workload API, which still expects a matching bearer token. The knowledge sources don’t document any way to turn that off via
auth.enabled: false
. What they do show is that similar
Unauthorized
/ 502 errors were fixed by ensuring all workload-related services share the same auth header and token: [[postgres 502 comment](https://github.com/airbytehq/airbyte/issues/65567); [file 502 comment](https://github.com/airbytehq/airbyte/issues/65574)] In
abctl
installs, users resolved it by:
Copy code
bash TOKEN=$(openssl rand -hex 24)  kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \   WORKLOAD_API_AUTH_HEADER=Authorization \   WORKLOAD_API_BEARER_TOKEN=$TOKEN  kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \   WORKLOAD_API_AUTH_HEADER=Authorization \   WORKLOAD_API_BEARER_TOKEN=$TOKEN
and in some cases also setting the same token on the worker/launcher and recreating the secret used by the Workload API. [[file 502 comment](https://github.com/airbytehq/airbyte/issues/65574); [1.8.3 502](https://github.com/airbytehq/airbyte/issues/66692)] However, the provided sources do not show: - How Helm 1.9.1 exposes
WORKLOAD_API_BEARER_TOKEN
/
WORKLOAD_API_AUTH_HEADER
as chart values, or - Any official way to “disable” workload auth through
auth.enabled: false
or other Helm values. So, based on what’s available: - Your current setting
auth.enabled: false
+
edition: community
does not control the workload API auth that is failing. - The underlying fix is to ensure the workload-api-server, workload-launcher, and worker share the same bearer token and header, but the sources do not document the exact Helm 1.9.1 configuration keys to do this. If you like, you can manually inspect the running Helm-based deployments (e.g.,
airbyte-workload-api-server
,
airbyte-workload-launcher
,
airbyte-worker
) with
kubectl describe deployment ...
and check whether
WORKLOAD_API_BEARER_TOKEN
and
WORKLOAD_API_AUTH_HEADER
are present and aligned; the knowledge sources just don’t provide the Helm-side configuration to change them.
e
what deployment info can I share with you so you can help me debug this? @kapa.ai
k
A message from kapa.ai
To debug this using only the available information, the most useful things you can share are: 1. Helm values relevant to auth & workload Specifically, the portions of your
values.yaml
that touch: -
global.auth.*
(you already shared
enabled: false
, but include the whole
auth:
block) [[auth docs](https://docs.airbyte.com/platform/next/deploying-airbyte/integrations/authentication)] - Any
global.jobs.kube.*
or
global.*
fields you set (these have been involved in other Helm issues). [[service account issue](https://github.com/airbytehq/airbyte/issues/49938); [workload pods sa](https://github.com/airbytehq/airbyte/issues/50979)] 2. Environment variables on the workload-related deployments For each of these deployments (names may vary slightly, but typically): -
airbyte-workload-api-server
-
airbyte-workload-launcher
-
airbyte-worker
(if present) Run (replace
<ns>
and deployment names as needed):
Copy code
bash    kubectl -n <ns> get deploy airbyte-workload-api-server -o yaml    kubectl -n <ns> get deploy airbyte-workload-launcher -o yaml    kubectl -n <ns> get deploy airbyte-worker -o yaml
Then share only the
env:
sections for the containers so we can see: -
WORKLOAD_API_BEARER_TOKEN
-
WORKLOAD_API_AUTH_HEADER
- Any
INTERNAL_API_AUTH_TOKEN
or similar This is exactly where misalignment was found in the 502/401 reports. [[postgres 502](https://github.com/airbytehq/airbyte/issues/65567); [1.8.3 502](https://github.com/airbytehq/airbyte/issues/66692)] 3. The namespace and deployment names you’re using Just the namespace and the exact deployment names for: - server / webapp - workload-api-server - workload-launcher This helps map your setup to the patterns in the
abctl
issues (which used
airbyte-abctl-*
names). [[1.8.3 502](https://github.com/airbytehq/airbyte/issues/66692); [file 502](https://github.com/airbytehq/airbyte/issues/65567)] 4. Relevant log snippets from workload-api-server You already shared the launcher stack trace. Please also share the corresponding logs from
airbyte-workload-api-server
around the same time (401/Unauthorized entries). That’s how users in the linked issues confirmed token/header problems. [[postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] If you provide: - The
auth:
section and other relevant Helm values, - The
env:
blocks for those deployments, - And the matching workload-api-server logs, then we can compare your configuration against the known working patterns from the 502/401 issues and see exactly where the mismatch is.