https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I have deployed airbyte using a Hel...
# replication-ask-ai
m
@kapa.ai I have deployed airbyte using a Helm chart all of the pods are running fine, whereas workload launcher pod is failing with the following error:
Copy code
Error: couldn't find key dataplane-client-id in Secret airbyte-ns/airbyte-auth-secrets
k
This is a known issue affecting several Airbyte versions (1.6.x and beyond). The error occurs because the 
dataplane-client-id
and 
dataplane-client-secret
keys are missing from the 
airbyte-auth-secrets
Kubernetes secret. [dataplane-client-id issue; workload-launcher failure] Root Cause The workload-launcher pod expects 
dataplane-client-id
and 
dataplane-client-secret
keys to exist in the 
airbyte-auth-secrets
secret. These are normally generated by the Airbyte bootloader during installation, but can be missing if: ā€¢ The 
airbyte-auth-secrets
secret was pre-created externally (e.g., to manage instance admin credentials), preventing the bootloader from writing the dataplane credentials into it. [workload-launcher failure] ā€¢ The bootloader failed to initialize the dataplane credentials properly during installation. [dataplane-client-id issue] Known Workarounds 1. If you pre-created `airbyte-auth-secrets`: Delete the existing secret from the cluster and let Airbyte recreate it automatically. This allows the bootloader to populate it with the required 
dataplane-client-id
and 
dataplane-client-secret
values. [workload-launcher failure] 2. If you need to keep your custom secret: Remove the following fields from your 
values.yaml
so Airbyte manages the secret itself:
Copy code
# Comment out or remove these:
   # global.auth.instanceAdmin.secretName
   # global.auth.instanceAdmin.emailSecretKey
   # global.auth.instanceAdmin.passwordSecretKey
[workload-launcher failure] 3. Manual population: Retrieve the 
dataplane-client-id
and 
dataplane-client-secret
values from the 
dataplane_client_credentials
table in the Airbyte database and manually add them to your secret. [dataplane-client-id issue] > Note: There is also a separate open bug (Helm chart v2.0.12) where duplicate 
DATAPLANE_CLIENT_ID_SECRET_KEY
entries in the ConfigMap can cause related issues. [duplicate ConfigMap keys] The most reliable fix reported by users is to delete the pre-existing 
airbyte-auth-secrets
secret and allow Airbyte to recreate it during deployment. [workload-launcher failure] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai If I delete the secret, will it be recreated by Airbyte itself?
k
Yes, based on user reports, deleting the existing 
airbyte-auth-secrets
secret allows Airbyte to recreate it automatically during deployment. The bootloader will then populate it with all the required keys, including 
dataplane-client-id
and 
dataplane-client-secret
. One user confirmed this outcome: > "After that, I deleted the existing secret from the cluster and deployed the changes. Airbyte was able to create the secret by itself (somehow it kept the same values for the instance admin keys), but also created the 
dataplane-client-id
and 
dataplane-client-secret
. And then the workload-launcher was able to launch." [workload-launcher failure] So yes ā€” Airbyte will recreate the secret, and it reportedly even preserves the instance admin email/password values in the process. The key requirement is that the secret must not be pre-existing and externally managed when Airbyte tries to write the dataplane credentials into it. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai I tried to delete the secret and resynced the pod for bootloader ran, new secret was created but again without 
dataplane-client-id
and 
dataplane-client-secret
. Whereas the bootloader pod succeeded.
k
This matches another known issue reported by users. Even when the bootloader pod succeeds, it may still regenerate/overwrite the 
dataplane-client-id
and 
dataplane-client-secret
without properly persisting them into the secret. [auth secret issue] Based on the knowledge sources, here are the remaining options: 1. Manually Populate the Secret from the Database Retrieve the values directly from the Airbyte database and add them manually to the secret: 1. Query the 
dataplane_client_credentials
table in your Airbyte database to get the 
dataplane-client-id
and 
dataplane-client-secret
values. 2. Add them to your secret:
Copy code
kubectl create secret generic airbyte-auth-secrets \
  --from-literal=dataplane-client-id='<value-from-db>' \
  --from-literal=dataplane-client-secret='<value-from-db>' \
  --namespace airbyte
[workload-launcher failure] 2. Remove External Secret Management from values.yaml If you are managing 
airbyte-auth-secrets
externally (e.g., for instance admin credentials), try removing those references from your 
values.yaml
entirely so Airbyte has full control over the secret:
Copy code
# Comment out or remove:
# global.auth.instanceAdmin.secretName
# global.auth.instanceAdmin.emailSecretKey
# global.auth.instanceAdmin.passwordSecretKey
[workload-launcher failure] 3. Upgrade to a Newer Version Some users reported that upgrading to 1.7.1 resolved the issue out of the box. [dataplane-client-id issue] ------------------------- Beyond these options, the knowledge sources do not contain further troubleshooting steps for this specific scenario. If none of these work, I'd recommend opening or following up on the relevant GitHub issues directly, as this appears to be an ongoing bug. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai after fixing that status for workload launcher changed from pending to running, but failing again with the following error:
Copy code
2026-03-06 12:10:10,480 [main]	ERROR	i.m.r.Micronaut(handleStartupException):349 - Error starting Micronaut server: Failed to initialize data-plane
java.lang.RuntimeException: Failed to initialize data-plane
	at io.airbyte.workload.launcher.authn.DataplaneIdentityService.initialize(DataplaneIdentityService.kt:56)
	at io.airbyte.workload.launcher.StartupApplicationEventListener.onApplicationEvent(StartupApplicationEventListener.kt:34)
	at io.airbyte.workload.launcher.StartupApplicationEventListener.onApplicationEvent(StartupApplicationEventListener.kt:20)
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:266)
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:226)
	at io.micronaut.http.server.netty.discovery.NettyServiceDiscovery.onStart(NettyServiceDiscovery.java:54)
	at io.micronaut.http.server.netty.discovery.$NettyServiceDiscovery$Definition$Exec.dispatch(Unknown Source)
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:456)
	at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:3944)
	at io.micronaut.aop.chain.AdapterIntroduction.intercept(AdapterIntroduction.java:87)
	at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:143)
	at io.micronaut.http.server.netty.discovery.NettyServiceDiscovery$ApplicationEventListener$onStart1$Intercepted.onApplicationEvent(Unknown Source)
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:266)
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:226)
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:692)
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:314)
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:112)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:81)
	at io.airbyte.workload.launcher.ApplicationKt.main(Application.kt:10)
Caused by: java.io.IOException: HTTP error: 500 Internal Server Error
	at io.airbyte.api.client.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:23)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at io.airbyte.api.client.UserAgentInterceptor.intercept(UserAgentInterceptor.kt:32)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at io.airbyte.api.client.auth.AirbyteAuthHeaderInterceptor.intercept(AirbyteAuthHeaderInterceptor.kt:28)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
	at dev.failsafe.CallImpl.execute(CallImpl.java:33)
	at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121)
	at io.airbyte.api.client.generated.DataplaneApi.initializeDataplaneWithHttpInfo(DataplaneApi.kt:2673)
	at io.airbyte.api.client.generated.DataplaneApi.initializeDataplane(DataplaneApi.kt:438)
	at io.airbyte.workload.launcher.authn.DataplaneIdentityService.initialize(DataplaneIdentityService.kt:41)
	... 18 common frames omitted
k
This error ā€” 
Failed to initialize data-plane
caused by 
HTTP error: 500 Internal Server Error
ā€” occurs when the workload-launcher successfully reads the 
dataplane-client-id
and 
dataplane-client-secret
from the secret, but then fails when trying to call the Airbyte API to register/initialize the dataplane. The 500 error comes from the server side. This is a known issue reported in multiple GitHub issues. [Multiple 2.0.0 issues; dataplane group mismatch] What This Error Means The workload-launcher is calling 
DataplaneApi.initializeDataplane
and receiving a 
500 Internal Server Error
from the Airbyte server. This typically indicates a mismatch or inconsistency between the dataplane credentials stored in the secret and what the Airbyte server (control plane) has registered in its database. Possible Causes & Steps to Try 1. Credential mismatch between secret and database: The `dataplane-client-id`/`dataplane-client-secret` in the Kubernetes secret may not match what's stored in the 
dataplane_client_credentials
table. Verify that the values in the secret (base64-decoded) exactly match the values in the database. [workload-launcher failure] 2. Stale/orphaned dataplane records: There may be stale records in the 
dataplane
and 
dataplane_client_credentials
tables. Some users attempted:
Copy code
DELETE FROM dataplane_client_credentials;
   DELETE FROM dataplane;
Then deleted the Kubernetes secret and let the bootloader recreate everything from scratch. Note that this did not work for all users. [workload-launcher failure] 3. Upgrade to 1.8.0+: A fix was merged that changes the bootloader behavior ā€” it now checks for valid credentials in the secret first before regenerating them, which should prevent credential mismatches. [auth secret issue] The knowledge sources do not contain a definitive resolution for the 500 error specifically. I'd recommend checking the server-side logs (e.g., the 
airbyte-server
pod) around the same timestamp to see what error the control plane is returning, as that will give more detail on why the initialization is failing on the server side. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai I had found 4 columns in 
dataplane_client_credentials
tables, and copied values for 
client_id
and 
client_secret
from there converted to base64 and added to the 
airbyte-auth-secrets
. What did I do wrong ?
k
Based on the knowledge sources, I can identify a likely issue with your approach. The Problem: Values Should NOT Be Re-encoded to Base64 When you copy values from the 
dataplane_client_credentials
table and add them to a Kubernetes secret, the encoding matters: ā€¢ Kubernetes secrets store values as base64-encoded strings internally. ā€¢ When you use 
kubectl create secret
with 
--from-literal
, Kubernetes handles the base64 encoding automatically. ā€¢ If you manually base64-encoded the values before adding them to the secret, the values would end up double-encoded, and the workload-launcher would read garbled/incorrect credentials. One user confirmed this exact nuance when validating credentials: > "curl ... `client_id`: `BASE64_DECODED_DATAPLANE_CLIENT_ID`" [workload-launcher failure] This implies the actual credential values used in API calls are the decoded (plain text) values ā€” not the base64 versions. What You Should Do Instead Use the raw plain-text values from the database and let 
kubectl
handle the encoding:
Copy code
kubectl patch secret airbyte-auth-secrets \
  --namespace airbyte-ns \
  --type merge \
  -p '{"stringData": {"dataplane-client-id": "<plain-text-client-id>", "dataplane-client-secret": "<plain-text-client-secret>"}}'
Or recreate using 
--from-literal
with plain-text values:
Copy code
kubectl create secret generic airbyte-auth-secrets \
  --from-literal=dataplane-client-id='<plain-text-value>' \
  --from-literal=dataplane-client-secret='<plain-text-value>' \
  --namespace airbyte-ns
Beyond this, the knowledge sources do not contain further detail on the exact column format in 
dataplane_client_credentials
. I'd recommend also checking whether the values in the database are already stored in an encoded format, which would require decoding them first before adding to the secret. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai Now getting this:
Copy code
2026-03-06 12:37:35,788 [Thread-2]	ERROR	i.a.a.c.a.DataplaneAccessTokenInterceptor(intercept):151 - Failed to obtain or add dataplane access token
java.lang.Exception: Token request failed with HTTP code 500
	at io.airbyte.api.client.auth.DataplaneAccessTokenInterceptor.fetchNewToken(DataplaneAccessTokenInterceptor.kt:110)
	at io.airbyte.api.client.auth.DataplaneAccessTokenInterceptor.getCachedToken$io_airbyte_airbyte_api_server_api(DataplaneAccessTokenInterceptor.kt:82)
	at io.airbyte.api.client.auth.DataplaneAccessTokenInterceptor.intercept(DataplaneAccessTokenInterceptor.kt:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
	at dev.failsafe.CallImpl.execute(CallImpl.java:33)
	at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121)
	at io.airbyte.workload.api.client.generated.WorkloadApi.workloadListWithHttpInfo(WorkloadApi.kt:4975)
	at io.airbyte.workload.api.client.generated.WorkloadApi.workloadList(WorkloadApi.kt:786)
	at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList$lambda$8(ClaimedProcessor.kt:122)
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
	at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList(ClaimedProcessor.kt:121)
	at io.airbyte.workload.launcher.ClaimedProcessor.retrieveAndProcess(ClaimedProcessor.kt:63)
	at io.airbyte.workload.launcher.StartupApplicationEventListener.onApplicationEvent$lambda$1(StartupApplicationEventListener.kt:39)
	at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)

2026-03-06 12:37:35,801 [Thread-2]	ERROR	i.a.w.l.StartupApplicationEventListener(onApplicationEvent$lambda$1):43 - Failed to retrieve and resume claimed workloads, exiting.
io.airbyte.workload.api.client.generated.infrastructure.ClientException: Client error : 401 Unauthorized {"message":"Unauthorized","logref":null,"path":null,"_links":{"self":{"href":"/api/v1/workload/list","templated":false,"profile":null,"deprecation":null,"title":null,"hreflang":null,"type":null,"name":null}},"_embedded":{"errors":[{"message":"Unauthorized","logref":null,"path":null,"_links":{},"_embedded":{}}]}}
	at io.airbyte.workload.api.client.generated.WorkloadApi.workloadList(WorkloadApi.kt:794)
	at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList$lambda$8(ClaimedProcessor.kt:122)
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
	at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList(ClaimedProcessor.kt:121)
	at io.airbyte.workload.launcher.ClaimedProcessor.retrieveAndProcess(ClaimedProcessor.kt:63)
	at io.airbyte.workload.launcher.StartupApplicationEventListener.onApplicationEvent$lambda$1(StartupApplicationEventListener.kt:39)
	at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)
k
This error shows two distinct problems happening in sequence: 1. 
Token request failed with HTTP code 500
 ā€” The workload-launcher is trying to obtain an access token using the `dataplane-client-id`/`dataplane-client-secret` credentials, but the token endpoint is returning a 500 error. 2. 
401 Unauthorized
 ā€” Because it couldn't get a valid token, the subsequent call to 
/api/v1/workload/list
is rejected as unauthorized. This is the same pattern reported by other users. [workload-launcher failure] What This Indicates The 500 error from the token endpoint strongly suggests that the credentials in the secret still don't match what the Airbyte server has registered in the 
dataplane_client_credentials
table. The server is failing to validate them, hence the 500, and the launcher then gets a 401 on subsequent calls. Steps to Verify 1. Validate the credentials directly against the token endpoint to confirm whether they are correct:
Copy code
curl -v --request POST \
  --url https://<YOUR_AIRBYTE_URL>/api/v1/dataplanes/token \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '{
    "client_id": "<plain-text-client-id-from-db>",
    "client_secret": "<plain-text-client-secret-from-db>"
  }'
If this returns an 
access_token
, the credentials are valid and the issue is elsewhere. If it returns a 500, the credentials in the database are mismatched or corrupted. [workload-launcher failure] 2. Check the server-side logs (
airbyte-server
pod) at the same timestamp to see the actual reason for the 500 ā€” the knowledge sources do not contain enough detail to determine the server-side cause without those logs. Most Reliable Recovery Path Based on what has worked for other users, the cleanest approach is: 1. Delete all records from 
dataplane_client_credentials
and 
dataplane
tables in the database. 2. Delete the 
airbyte-auth-secrets
Kubernetes secret. 3. Let the bootloader recreate everything from scratch on the next deployment. However, note that some users reported this did not work reliably on 1.6.x versions. [workload-launcher failure] A fix for the underlying bootloader behavior (checking for valid credentials before regenerating) was merged and is available in version 1.8.0+. [auth secret issue] If upgrading is an option, that may be the most reliable resolution. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai To verify when curled to my Airbyte instance got the following response:
Copy code
* upload completely sent off: 170 bytes
< HTTP/2 500 
< date: Fri, 06 Mar 2026 14:12:36 GMT
< content-type: application/json
< content-length: 174
< server: nginx
< 
* Connection #0 to host <MY_AIRBYTE_URL>io left intact
{"message":"Internal Server Error: Invalid clientSecret provided.","exceptionClassName":"java.lang.IllegalArgumentException","exceptionStack":[],"rootCauseExceptionStack":[]}%
k
The response confirms the root cause: 
Invalid clientSecret provided
. This means the 
dataplane-client-secret
value currently in your 
airbyte-auth-secrets
Kubernetes secret does not match what the Airbyte server has stored in the 
dataplane_client_credentials
table. What Likely Went Wrong When you copied the 
client_secret
from the database and base64-encoded it before adding it to the secret, the value became double-encoded. Even after correcting that, there may still be a mismatch ā€” for example, if the value in the database is stored in a hashed or encoded format rather than plain text. Recommended Recovery Steps The cleanest path reported by users is to wipe the dataplane records and let the bootloader regenerate everything: 1. Connect to the Airbyte database:
Copy code
kubectl exec -it -n <your-namespace> airbyte-db-0 -- /bin/bash
   psql -U airbyte db-airbyte
2. Delete the stale dataplane records:
Copy code
DELETE FROM dataplane_client_credentials;
   DELETE FROM dataplane;
   \q
3. Delete the Kubernetes secret:
Copy code
kubectl delete secret airbyte-auth-secrets -n <your-namespace>
4. Redeploy/restart so the bootloader runs again and regenerates the credentials from scratch, writing the new values into the secret. [workload-launcher failure] This ensures the credentials in the secret and the database are in sync because they are generated together in the same bootloader run. > Important: If you are on a version older than 1.8.0, the bootloader may regenerate the dataplane credentials on every deploy, causing the mismatch to recur. Upgrading to 1.8.0+ includes a fix where the bootloader checks for valid existing credentials before regenerating them. [auth secret issue] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai redeployed the airbyte but it stuck at the smae workload launcher issue where its not able to find the dataplane client id.
k
Based on the knowledge sources, this is a persistent issue that several users have encountered even after attempting the database cleanup and secret deletion steps. [workload-launcher failure] What's Likely Happening The bootloader is succeeding but not writing 
dataplane-client-id
and 
dataplane-client-secret
into the newly created secret. This can happen when the secret is being recreated externally (e.g., by Helm) before the bootloader has a chance to populate it, or when external secret management is interfering. Key Things to Check 1. Are you using an externally managed secret? If your 
values.yaml
still references a pre-existing secret for instance admin credentials, this is the most likely culprit:
Copy code
# If any of these are set, comment them out:
   # global.auth.instanceAdmin.secretName
   # global.auth.instanceAdmin.emailSecretKey
   # global.auth.instanceAdmin.passwordSecretKey
One user confirmed that removing these references and deleting the secret allowed Airbyte to fully manage it ā€” including writing the dataplane credentials. [workload-launcher failure] 2. Verify the secret contents after bootloader runs:
Copy code
kubectl get secret airbyte-auth-secrets -n <your-namespace> -o jsonpath='{.data}' | jq 'keys'
Check whether 
dataplane-client-id
and 
dataplane-client-secret
keys are present at all. 3. Check bootloader logs carefully for any errors related to dataplane initialization ā€” a successful exit code doesn't necessarily mean the dataplane credentials were written successfully. [auth secret issue] Most Reliable Fix Reported The user who resolved this confirmed the following sequence worked: ā€¢ Removed all 
global.auth.instanceAdmin
references from 
values.yaml
ā€¢ Deleted the existing 
airbyte-auth-secrets
secret ā€¢ Redeployed ā€” Airbyte recreated the secret with all required keys including 
dataplane-client-id
and 
dataplane-client-secret
[workload-launcher failure] If you are still on a version below 1.8.0, upgrading is strongly recommended as it includes a fix for the bootloader's credential regeneration behavior. [auth secret issue] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai After upgrading to 1.7.0 this has resolved the issue, workload launcher is up. But I have noticed that Dagster user deployment is not able to make cconenction to Airbyte and throwing the following error:
Copy code
Request to Airbyte API failed: 401 Client Error: Unauthorized for url: <http://airbyte-airbyte-webapp-svc.data-services.svc.cluster.local:80/api/v1/workspaces/list>
k
The 401 error from Dagster when calling the Airbyte API is an authentication issue. Starting from recent Airbyte versions, the API requires authentication, and Dagster needs valid credentials to connect. What's Happening Airbyte now has built-in authentication enabled by default. The Dagster 
airbyte_resource
is making unauthenticated requests to the Airbyte API, which are being rejected with 
401 Unauthorized
. [Airbyte Authentication] Options to Fix This Option 1: Disable Airbyte Authentication (if acceptable) If you have another authentication layer (e.g., a proxy with SSO), you can disable Airbyte's built-in auth by adding the following to your `values.yaml`:
Copy code
global:
  auth:
    enabled: false
Then apply it:
Copy code
abctl local install --values ./values.yaml
[Airbyte Authentication] Option 2: Retrieve Credentials and Configure Dagster Retrieve your Airbyte credentials:
Copy code
kubectl get secret airbyte-auth-secrets -n <YOUR_NAMESPACE> -o yaml
The values are base64-encoded. Decode them:
Copy code
echo '<base64-encoded-value>' | base64 -d
[Airbyte Authentication] Then configure your Dagster 
airbyte_resource
to pass the credentials. The Dagster Airbyte resource supports 
username
and 
password
parameters for basic auth. [Dagster Integration] Option 3: Also Check the Endpoint Note that your Dagster integration is calling 
/api/v1/workspaces/list
on the webapp service, not the API server directly. Ensure Dagster is pointed at the correct Airbyte API service endpoint in your cluster. [Dagster Integration] The knowledge sources do not contain specific details on how to pass credentials in the Dagster 
airbyte_resource
configuration beyond the 
host
and 
port
parameters shown in the docs. I'd recommend checking the Dagster Airbyte integration docs directly for credential configuration options. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai In dagster user deployment I have provided the following as the Airbyte host:
Copy code
airbyte-airbyte-webapp-svc.data-services.svc.cluster.local
Whihc is the internal dns for airbyte webapp service.
k
Based on the knowledge sources available, I can see that when setting up Dagster to connect to Airbyte, the tutorials consistently use the following environment variable pattern:
Copy code
export AIRBYTE_PASSWORD=password
And the Dagster UI is accessed at 
http://127.0.0.1:3000
, with Airbyte running locally. [E-commerce Analytics; Developer Productivity Stack] However, the knowledge sources do not contain specific guidance on: ā€¢ Whether Dagster should connect to the webapp service vs. the API server service in a Kubernetes deployment ā€¢ The correct internal DNS endpoint to use for Dagster ā†’ Airbyte communication in a cluster setup What I can say is that the 401 error you're seeing suggests an authentication problem, not necessarily a routing problem. The webapp service may not be the correct endpoint for API calls ā€” typically the 
airbyte-server
service handles API requests. I'd recommend: 1. Trying the 
airbyte-server
service instead (e.g., 
airbyte-airbyte-server-svc.data-services.svc.cluster.local
) as the host for Dagster. 2. Ensuring the 
AIRBYTE_PASSWORD
environment variable is set to the correct password retrieved from your secret. [Airbyte Authentication] Beyond this, the knowledge sources do not contain enough detail to confirm the exact correct service endpoint for Dagster in a Helm-deployed Kubernetes setup. I'd recommend checking the Dagster Airbyte integration documentation directly for Kubernetes-specific configuration. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai after changing the airbyte host URL from 
airbyte-airbyte-webapp-svc.data-services.svc.cluster.local
to ``airbyte-airbyte-server-svc.data-services.svc.cluster.local``, it started getting the following error:
Copy code
ERROR - Request to Airbyte API failed: HTTPConnectionPool(host='airbyte-airbyte-server-svc.data-services.svc.cluster.local', port=80): Max retries exceeded with url: /api/v1/workspaces/list (Caused by ConnectTimeoutError(<HTTPConnection(host='airbyte-airbyte-server-svc.data-services.svc.cluster.local', port=80) at 0x7f6a3968a240>, 'Connection to airbyte-airbyte-server-svc.data-services.svc.cluster.local timed out. (connect timeout=15)'))
k
The timeout error indicates that Dagster can resolve the DNS name but cannot connect on port 80. This is a port mismatch ā€” the 
airbyte-airbyte-server-svc
does not listen on port 80; it listens on port 8001. From the ingress configuration in the Airbyte docs, the server service is mapped as follows:
Copy code
- backend:
    service:
      name: airbyte-airbyte-server-svc
      port:
        number: 8001
  path: /
  pathType: Prefix
[Ingress config] What to Fix Update your Dagster 
airbyte_resource
configuration to use port 8001 instead of port 80:
Copy code
my_airbyte_resource = airbyte_resource.configured(
    {
        "host": "airbyte-airbyte-server-svc.data-services.svc.cluster.local",
        "port": "8001",
    }
)
[Dagster Integration] This should resolve the connection timeout. If you still get a 401 after connecting successfully, you'll also need to provide valid credentials (username/password) in the Dagster resource configuration, as Airbyte 1.7.0 has authentication enabled by default. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai Now started getting this:
Copy code
ERROR - Request to Airbyte API failed: 403 Client Error: Forbidden for url: <http://airbyte-airbyte-server-svc.data-services.svc.cluster.local:8001/api/v1/workspaces/list>
k
The connection is now working (no more timeout), but Airbyte is rejecting the request with 401 Unauthorized because the Dagster resource is not passing credentials. As of Airbyte 1.7.0, authentication is enabled by default. You need to provide valid credentials to the Dagster 
airbyte_resource
. Option 1: Pass Credentials in the Dagster Resource First, retrieve your Airbyte credentials from the Kubernetes secret:
Copy code
kubectl get secret airbyte-auth-secrets -n <YOUR_NAMESPACE> -o yaml
The values are base64-encoded. Decode the password:
Copy code
echo '<base64-encoded-instance-admin-password>' | base64 -d
[Airbyte Authentication] Then update your Dagster resource configuration with the credentials. The 
airbyte_resource
accepts 
username
and 
password
parameters:
Copy code
my_airbyte_resource = airbyte_resource.configured(
    {
        "host": "airbyte-airbyte-server-svc.data-services.svc.cluster.local",
        "port": "8001",
        "username": "airbyte",
        "password": {"env": "AIRBYTE_PASSWORD"},
    }
)
[Dagster Integration] Option 2: Disable Airbyte Authentication If you have another authentication layer in place, you can disable Airbyte's built-in auth by adding the following to your 
values.yaml
and redeploying:
Copy code
global:
  auth:
    enabled: false
[Airbyte Authentication] The knowledge sources do not explicitly document all available parameters for the Dagster 
airbyte_resource
ā€” I'd recommend checking the Dagster Airbyte integration docs directly to confirm the exact parameter names for passing credentials. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
The error has progressed from 401 Unauthorized to 403 Forbidden, which means authentication is now succeeding (credentials are being accepted), but the authenticated user/application does not have permission to access the 
/api/v1/workspaces/list
endpoint. What This Means A 403 typically indicates that the credentials being used are valid, but the associated user or application lacks the required permissions for the requested resource. In Airbyte, access tokens represent individual users and carry the same permissions as that user. [API access config] Things to Check 1. Are you using the correct credentials? The 
instance-admin
credentials from 
airbyte-auth-secrets
should have full admin access. Verify you are using the 
instance-admin-client-id
and 
instance-admin-client-secret
(or the decoded 
instance-admin-password
) rather than the `dataplane-client-id`/`dataplane-client-secret`, which are scoped specifically for the workload launcher ā€” not for general API access. Decode and verify which credentials you're using:
Copy code
kubectl get secret airbyte-auth-secrets -n <YOUR_NAMESPACE> -o yaml
   echo '<base64-encoded-value>' | base64 -d
[Airbyte Authentication] 2. Use the correct API path. The 
/api/v1/workspaces/list
path is an internal API path. The public API endpoint for listing workspaces is 
/api/public/v1/workspaces
. [API access config] 3. Consider using an Application token instead of basic auth. The recommended approach for programmatic API access is to create an Application in the Airbyte UI, obtain a Bearer token, and use that: ā—¦ Go to User settings > Applications in the Airbyte UI ā—¦ Create an application to get a 
client_id
and 
client_secret
ā—¦ Exchange them for an access token at 
POST
YOUR_AIRBYTE_URL`/api/v1/applications/token` ā—¦ Use the token as a Bearer token in API requests [API access config] The knowledge sources do not contain specific details on how the Dagster 
airbyte_resource
maps its credentials to Airbyte's authentication system internally, so I'd recommend checking the Dagster Airbyte integration docs to confirm exactly how it passes credentials to the API. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
5 Views
Open in Slack
PreviousNext
Powered by