Eduardo Ferreira
12/02/2025, 12:47 PMkapa.ai
12/02/2025, 12:47 PMio.airbyte.api.client.ApiException: Unauthorized from airbyte-workload-launcher means its requests to the Workload API are being rejected with 401, even if you believe “auth is disabled”. From similar issues, this almost always comes down to the Workload API bearer token / header not matching between the launcher and the Workload API server (and sometimes worker). [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502 comment](https://github.com/airbytehq/airbyte/issues/65567)] Key points from the reports: - In local abctl installs, the fix was to recreate the secret that holds the bearer token used by the Workload API and worker: [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574)] bash KUBECONFIG="${HOME}/.airbyte/abctl/abctl.kubeconfig" NAMESPACE="airbyte-abctl" SECRET_NAME="airbyte-abctl-workload-api-secrets" WORKLOAD_TOKEN=$(openssl rand -hex 32) kubectl delete secret "$SECRET_NAME" -n "$NAMESPACE" --ignore-not-found --kubeconfig "$KUBECONFIG" kubectl create secret generic "$SECRET_NAME" \ --from-literal=bearer-token="$WORKLOAD_TOKEN" \ -n "$NAMESPACE" \ --kubeconfig "$KUBECONFIG" - In another case, all Workload-related deployments had to share the same WORKLOAD_API_AUTH_HEADER and `WORKLOAD_API_BEARER_TOKEN`: [[postgres 502 comment](https://github.com/airbytehq/airbyte/issues/65567)] bash TOKEN=$(openssl rand -hex 24) kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \ WORKLOAD_API_AUTH_HEADER=Authorization \ WORKLOAD_API_BEARER_TOKEN=$TOKEN kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \ WORKLOAD_API_AUTH_HEADER=Authorization \ WORKLOAD_API_BEARER_TOKEN=$TOKEN kubectl -n airbyte-abctl set env deployment/airbyte-workload-launcher \ WORKLOAD_API_AUTH_HEADER=Authorization \ WORKLOAD_API_BEARER_TOKEN=$TOKEN Given your error is in workloadList from the launcher, the concrete steps I’d recommend: 1. Identify the namespace and deployment names you’re using (Helm vs abctl may differ). 2. Inspect env vars on: - airbyte-workload-api-server - airbyte-workload-launcher - (and worker if present) and confirm WORKLOAD_API_AUTH_HEADER and WORKLOAD_API_BEARER_TOKEN are identical. 3. If in doubt, generate a new token and set it on all of them as in the snippet above, then restart those pods. If you truly have a mode where auth is supposed to be disabled entirely, the knowledge sources do not document how to configure that or how the launcher behaves in that case, so I can’t reliably describe additional flags or settings beyond ensuring the tokens line up.Eduardo Ferreira
12/02/2025, 12:54 PMkapa.ai
12/02/2025, 12:54 PMkapa.ai
12/02/2025, 12:55 PMUnauthorized errors between the workload launcher and the Workload API are the same class of issue that people hit with abctl in 1.8.x: the Workload API auth header/token are either missing or not consistent between components. However, all concrete remediation steps in the knowledge sources are for abctl / local deployments (airbyte-abctl-* deployments), not the Helm chart you’re using. [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502](https://github.com/airbytehq/airbyte/issues/65567); [1.8.3 502](https://github.com/airbytehq/airbyte/issues/66692)] Those sources show: - The failure mode: io.airbyte.api.client.ApiException: Unauthorized from workload-related components when calling the Workload API. [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] - The workaround: manually setting the same WORKLOAD_API_BEARER_TOKEN (and sometimes WORKLOAD_API_AUTH_HEADER) environment variables on the workload API server and worker, then restarting those deployments. [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] What’s not present in the knowledge sources: - Any Helm 1.9.1–specific configuration knobs (values.yaml keys) for disabling workload auth or wiring those tokens for Helm deployments. - Any official doc or issue explicitly describing how to fix this Unauthorized error on Helm 1.9.1. Because of that, I cannot provide a Helm-1.9.1-specific, chart-values-based fix from these sources. What you can take from the existing information: - The root cause is almost certainly that the Helm deployment of: - airbyte-workload-api-server - airbyte-workload-launcher (and possibly airbyte-worker) does not have aligned WORKLOAD_API_BEARER_TOKEN / WORKLOAD_API_AUTH_HEADER, similar to the abctl cases. [[postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] - A likely workaround (conceptually) would be to ensure those env vars are set to the same values across those deployments and then restart, just as users did with abctl. [[file 502 issue](https://github.com/airbytehq/airbyte/issues/65574); [postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] But since there is no documentation here on how Helm 1.9.1 exposes or manages those variables, I can’t reliably tell you: - Which exact Helm values to set, or - Whether there is a Helm-native way to “disable” this auth for 1.9.1. So, based strictly on the available sources: - The problem you’re seeing on Helm 1.9.1 is consistent with known Workload API auth/token mismatches. - The only documented fixes are for `abctl`/local and involve manually aligning WORKLOAD_API_BEARER_TOKEN (and optionally the header) between the Workload API and worker/launcher. - The knowledge sources do not contain enough Helm-specific information for 1.9.1 to give an exact configuration snippet or official fix.Eduardo Ferreira
12/02/2025, 2:39 PMauth:
enabled: false
edition: communitykapa.ai
12/02/2025, 2:39 PMkapa.ai
12/02/2025, 2:40 PMauth.enabled: false in the Helm values only disables user-facing auth; it does not disable the internal authentication between the workload components (workload-launcher, workload-api-server, worker). The Unauthorized you see is from the launcher calling the Workload API, which still expects a matching bearer token. The knowledge sources don’t document any way to turn that off via auth.enabled: false. What they do show is that similar Unauthorized / 502 errors were fixed by ensuring all workload-related services share the same auth header and token: [[postgres 502 comment](https://github.com/airbytehq/airbyte/issues/65567); [file 502 comment](https://github.com/airbytehq/airbyte/issues/65574)] In abctl installs, users resolved it by: bash TOKEN=$(openssl rand -hex 24) kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \ WORKLOAD_API_AUTH_HEADER=Authorization \ WORKLOAD_API_BEARER_TOKEN=$TOKEN kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \ WORKLOAD_API_AUTH_HEADER=Authorization \ WORKLOAD_API_BEARER_TOKEN=$TOKEN and in some cases also setting the same token on the worker/launcher and recreating the secret used by the Workload API. [[file 502 comment](https://github.com/airbytehq/airbyte/issues/65574); [1.8.3 502](https://github.com/airbytehq/airbyte/issues/66692)] However, the provided sources do not show: - How Helm 1.9.1 exposes WORKLOAD_API_BEARER_TOKEN / WORKLOAD_API_AUTH_HEADER as chart values, or - Any official way to “disable” workload auth through auth.enabled: false or other Helm values. So, based on what’s available: - Your current setting auth.enabled: false + edition: community does not control the workload API auth that is failing. - The underlying fix is to ensure the workload-api-server, workload-launcher, and worker share the same bearer token and header, but the sources do not document the exact Helm 1.9.1 configuration keys to do this. If you like, you can manually inspect the running Helm-based deployments (e.g., airbyte-workload-api-server, airbyte-workload-launcher, airbyte-worker) with kubectl describe deployment ... and check whether WORKLOAD_API_BEARER_TOKEN and WORKLOAD_API_AUTH_HEADER are present and aligned; the knowledge sources just don’t provide the Helm-side configuration to change them.Eduardo Ferreira
12/02/2025, 5:50 PMkapa.ai
12/02/2025, 5:50 PMkapa.ai
12/02/2025, 5:51 PMvalues.yaml that touch: - global.auth.* (you already shared enabled: false, but include the whole auth: block) [[auth docs](https://docs.airbyte.com/platform/next/deploying-airbyte/integrations/authentication)] - Any global.jobs.kube.* or global.* fields you set (these have been involved in other Helm issues). [[service account issue](https://github.com/airbytehq/airbyte/issues/49938); [workload pods sa](https://github.com/airbytehq/airbyte/issues/50979)] 2. Environment variables on the workload-related deployments For each of these deployments (names may vary slightly, but typically): - airbyte-workload-api-server - airbyte-workload-launcher - airbyte-worker (if present) Run (replace <ns> and deployment names as needed): bash kubectl -n <ns> get deploy airbyte-workload-api-server -o yaml kubectl -n <ns> get deploy airbyte-workload-launcher -o yaml kubectl -n <ns> get deploy airbyte-worker -o yaml Then share only the env: sections for the containers so we can see: - WORKLOAD_API_BEARER_TOKEN - WORKLOAD_API_AUTH_HEADER - Any INTERNAL_API_AUTH_TOKEN or similar This is exactly where misalignment was found in the 502/401 reports. [[postgres 502](https://github.com/airbytehq/airbyte/issues/65567); [1.8.3 502](https://github.com/airbytehq/airbyte/issues/66692)] 3. The namespace and deployment names you’re using Just the namespace and the exact deployment names for: - server / webapp - workload-api-server - workload-launcher This helps map your setup to the patterns in the abctl issues (which used airbyte-abctl-* names). [[1.8.3 502](https://github.com/airbytehq/airbyte/issues/66692); [file 502](https://github.com/airbytehq/airbyte/issues/65567)] 4. Relevant log snippets from workload-api-server You already shared the launcher stack trace. Please also share the corresponding logs from airbyte-workload-api-server around the same time (401/Unauthorized entries). That’s how users in the linked issues confirmed token/header problems. [[postgres 502](https://github.com/airbytehq/airbyte/issues/65567)] If you provide: - The auth: section and other relevant Helm values, - The env: blocks for those deployments, - And the matching workload-api-server logs, then we can compare your configuration against the known working patterns from the 502/401 issues and see exactly where the mismatch is.