Eduardo Ferreira
11/24/2025, 9:28 PMError: couldn't find key dataplane-client-id in Secret airbyte/airbyte-auth-secrets. What values should this be set to? Note that I'm using the oss airbyte core version. I was previously on the chart 1.5.x and we didnt have that requirement. dataplane is not cited in any of the migration docskapa.ai
11/24/2025, 9:28 PMairbyte-auth-secrets Secret, and the workload-launcher/init containers fail if those keys are missing. [[1.6.0 issue](https://github.com/airbytehq/airbyte/issues/62468); [1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] However, for OSS single-cluster (“no separate data plane”) there is no documented, user-facing value you’re supposed to provide for dataplane-client-id / dataplane-client-secret, and the official docs and migration guides you referenced indeed don’t describe any manual setup for that. The existing issues show: - The workload-launcher/init containers expect keys like dataplane-client-id and dataplane-client-secret in airbyte-auth-secrets, and fail with exactly your error when they’re missing. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587); [1.6.0 chart issue](https://github.com/airbytehq/airbyte/issues/62468)] - When users pre‑create airbyte-auth-secrets (e.g., just for instance admin credentials), the bootloader may not populate or may overwrite the data plane keys, leading to the failure. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587); [bootloader overwrite report](https://github.com/airbytehq/airbyte/issues/64159)] - A workaround that made things work was not managing airbyte-auth-secrets externally: remove your custom secret and any Helm values that point to it, let Airbyte/bootloader create and populate that Secret itself; this also caused the data plane credentials to be created and the workload-launcher to become healthy. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] In your case (OSS, Helm v1, 2.0.1): - The knowledge sources do not document what concrete values to set for dataplane-client-id / dataplane-client-secret in an OSS-only install, nor a supported way to hand‑craft them. - For Enterprise Flex / separate data plane, the docs clearly say you must use the client id/secret issued by the control plane and store them in a secret (e.g. airbyte-config-secrets) under keys like DATA_PLANE_CLIENT_ID / DATA_PLANE_CLIENT_SECRET, then reference them in values.yaml. [[Enterprise-flex data plane](https://docs.airbyte.com/platform/next/enterprise-flex/data-plane#step-3)] That model doesn’t apply directly to OSS core. Given the above, the only behavior we can rely on from the sources is: 1. If you have a manually managed airbyte-auth-secrets, that is likely causing the error. 2. Deleting that secret and the corresponding values.yaml references and allowing the bootloader to recreate/populate it has resolved this class of errors for other users. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] Because the knowledge sources do not specify how to correctly hand‑set these values for OSS, I can’t tell you “use X or Y string”; they appear to be generated and stored by the platform itself, and manually setting them is fragile (and in some versions, overwritten by bootloader anyway). [[bootloader overwrite report](https://github.com/airbytehq/airbyte/issues/64159); [62468 comments](https://github.com/airbytehq/airbyte/issues/62468)] So, with what we have in the sources: - There is no documented, stable value you should set for dataplane-client-id / dataplane-client-secret for OSS Helm v1. - The recommended practical step from existing issues is: - Remove your custom airbyte-auth-secrets from the cluster; - Remove or comment out any Helm values that explicitly set auth.instanceAdmin.secretName, emailSecretKey, passwordSecretKey to that Secret so the chart can manage it itself; [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] - Re‑deploy so the bootloader creates and populates airbyte-auth-secrets, including the data plane keys it expects. If you must keep managing instance admin credentials via your own secret, the knowledge sources do not provide a supported pattern for doing that and providing fixed dataplane credentials in OSS; several users report conflicts and inconsistent behavior in exactly that setup. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587); [62468 comments](https://github.com/airbytehq/airbyte/issues/62468)]Eduardo Ferreira
11/24/2025, 9:33 PMkapa.ai
11/24/2025, 9:33 PMkapa.ai
11/24/2025, 9:34 PMairbyte-auth-secrets is missing the keys dataplane-client-id / dataplane-client-secret. [[1.6.0 chart issue](https://github.com/airbytehq/airbyte/issues/62468); [1.6.x abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] - When users pre‑create and manage airbyte-auth-secrets themselves (typically just for instance-admin-email / instance-admin-password), the bootloader and workload pods cannot reliably populate or use the dataplane keys, causing your exact error. [[1.6.x abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] - A working pattern reported in those issues is to let the chart/bootloader manage airbyte-auth-secrets entirely, instead of managing it yourself. Deleting the manually created secret and the related Helm values allows Airbyte to recreate the secret and add dataplane-client-id and dataplane-client-secret, after which the workload-launcher runs successfully. [[1.6.x abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] The knowledge sources do not document: - Any explicit values you should set for dataplane-client-id / dataplane-client-secret in an OSS Helm deployment. - A supported way to hand‑craft those values for OSS. In fact, there are reports that even when users manually set them in the secret, the bootloader overwrites them. [[bootloader overwrite](https://github.com/airbytehq/airbyte/issues/64159); [62468 comments](https://github.com/airbytehq/airbyte/issues/62468)] Because of that, I can’t reliably tell you “set them to X” from the docs: they appear to be generated and managed internally, not user‑provided values in OSS. Given you are on Helm: 1. If you currently define a custom auth secret in values.yaml, e.g.: yaml auth: enabled: true instanceAdmin: secretName: airbyte-auth-secrets emailSecretKey: instance-admin-email passwordSecretKey: instance-admin-password this configuration is exactly the one that has triggered the error for others. [[1.6.x abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] 2. The only workaround clearly shown to work in the sources is: - Delete the existing airbyte-auth-secrets from your namespace. - Remove or comment out the auth.instanceAdmin.secretName / emailSecretKey / passwordSecretKey overrides from your Helm values so the chart can manage that secret itself. [[1.6.x abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] - Re‑deploy the chart so the bootloader recreates airbyte-auth-secrets and injects dataplane-client-id / dataplane-client-secret. The sources do not describe a safe, documented way (for OSS + Helm) to keep your own airbyte-auth-secrets and manually supply dataplane values, so I can’t recommend a specific values.yaml pattern for that scenario based on the available information.Eduardo Ferreira
11/24/2025, 9:35 PMkapa.ai
11/24/2025, 9:39 PMkapa.ai
11/24/2025, 9:40 PMinstance-admin-email etc. when you manage secrets externally) is not clearly documented in the official docs, and the only concrete evidence we have comes from GitHub issues. From those: - When users pre‑create and externally manage the airbyte-auth-secrets Secret (for example, via AWS Secrets Manager + ExternalSecrets), Airbyte’s bootloader still tries to manage data‑plane–related keys (dataplane-client-id, dataplane-client-secret) inside that same secret. [[1.6.0 dataplane issue](https://github.com/airbytehq/airbyte/issues/62468); [auth secret overwrite](https://github.com/airbytehq/airbyte/issues/64159)] - At least in some versions, the bootloader overwrites credentials in the auth secret when it initializes the dataplane. This is why a user explicitly asked: “is it possible to define dataplane-client credentials without rewriting the auth secret?” [[1.6.0 dataplane issue](https://github.com/airbytehq/airbyte/issues/62468); [auth secret overwrite](https://github.com/airbytehq/airbyte/issues/64159)] - Another user reported that when they had airbyte-auth-secrets pre‑created and managed externally, the secret was not being updated with the dataplane keys, even though the role was configured to allow it. As a workaround they: - Commented out the global.auth.instanceAdmin.secretName/emailSecretKey/passwordSecretKey overrides in Helm (which pointed to their managed secret), and - Deleted the existing airbyte-auth-secrets. After that, Airbyte re‑created airbyte-auth-secrets itself, “somehow it kept the same values for the instance admin keys, but also created the dataplane-client-id and dataplane-client-secret,” and the system became healthy. [[abctl 1.6.0 issue](https://github.com/airbytehq/airbyte/issues/58587)] However, there is no explicit documentation of: - A supported pattern for using AWS Secrets Manager / ExternalSecrets to manage instance-admin-email and instance-admin-password while also letting Airbyte manage dataplane keys in the same secret. - Any guarantee that the bootloader will always preserve externally provided instance-admin-* keys when it (re)creates or updates airbyte-auth-secrets. So, based on what we do know: 1. If you keep managing airbyte-auth-secrets entirely via AWS Secrets Manager: - The knowledge sources do not state that your existing instance-admin-email and instance-admin-password will be preserved. - There is evidence that bootloader may overwrite or fail to update keys in that secret depending on version and initialization flow. [[auth secret overwrite](https://github.com/airbytehq/airbyte/issues/64159); [abctl 1.6.0 issue](https://github.com/airbytehq/airbyte/issues/58587)] 2. If you let Airbyte/bootloader create and manage `airbyte-auth-secrets`: - A user observed that their instance admin keys “somehow” stayed the same when the secret was re‑created, but that’s an observation, not a documented contract. [[abctl 1.6.0 issue](https://github.com/airbytehq/airbyte/issues/58587)] - This is currently the only configuration in the sources that clearly results in valid dataplane keys plus working workload‑launcher. Because the docs and issues do not describe a stable, supported interaction between ExternalSecrets/AWS Secrets Manager and the airbyte-auth-secrets bootloader logic, I can’t confidently say that your externally managed instance-admin-email and related fields will be “safe” or untouched. The safest interpretation from the available information is: - airbyte-auth-secrets is expected to be platform‑managed rather than fully controlled by an external secret manager. - Using AWS Secrets Manager to manage other Airbyte secrets (for connectors etc.) is documented and supported, but `airbyte-auth-secrets`’s internal structure and lifecycle are not documented as user‑configurable. [[Secret management values](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets#values)] If you must keep using AWS Secrets Manager for admin credentials, the current knowledge sources don’t provide a clear, supported way to do that without risking conflicts with bootloader’s management of the same secret.Eduardo Ferreira
11/24/2025, 9:54 PMkapa.ai
11/24/2025, 9:54 PMkapa.ai
11/24/2025, 9:55 PMDataplaneInitializer that creates and/or updates dataplane client credentials and writes them into Kubernetes secrets. [[bootloader DataplaneInitializer](https://github.com/airbytehq/airbyte/issues/62468); [bootloader overwrite](https://github.com/airbytehq/airbyte/issues/64159)] - For Enterprise Flex data planes, the control plane issues DATA_PLANE_CLIENT_ID and DATA_PLANE_CLIENT_SECRET, and you store them in a K8s secret like airbyte-config-secrets (or equivalent), then reference them from values.yaml. [[Enterprise data plane](https://docs.airbyte.com/platform/next/enterprise-flex/data-plane#step-3)] - In OSS Helm v1, dataplane-related values are embedded into the auth / workload config, and several issues show that DATAPLANE_CLIENT_ID / DATAPLANE_CLIENT_SECRET (or dataplane-client-id keys) are injected via airbyte-auth-secrets. [[1.6.0 dataplane issue](https://github.com/airbytehq/airbyte/issues/62468); [config map keys](https://github.com/airbytehq/airbyte/issues/66546)] 2. Location in OSS Helm installs - Errors like couldn't find key dataplane-client-id in Secret <ns>/airbyte-auth-secrets show that, in your OSS + Helm setup, the bootloader and workload-launcher expect these values inside the airbyte-auth-secrets K8s Secret. [[1.6.x dataplane error](https://github.com/airbytehq/airbyte/issues/58587); [1.6.0 chart issue](https://github.com/airbytehq/airbyte/issues/62468)] - When users do not pre-create that secret and let Airbyte/bootloader manage it, it ends up containing dataplane-client-id and dataplane-client-secret, and the system works. [[1.6.x dataplane error](https://github.com/airbytehq/airbyte/issues/58587)] 3. Interaction with external secret managers is not documented - The docs describe using external secret managers (AWS Secrets Manager, etc.) for general secrets, but they do not document a pattern for syncing the auto-generated dataplane credentials into an external manager. [[Secret management values](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets#values)] - Users who pre‑create/manage airbyte-auth-secrets externally (including dataplane keys) have run into problems: the bootloader may overwrite or fail to populate those keys. [[auth secret overwrite](https://github.com/airbytehq/airbyte/issues/64159); [1.6.x dataplane error](https://github.com/airbytehq/airbyte/issues/58587); [62468 comments](https://github.com/airbytehq/airbyte/issues/62468)] - There is no doc or issue comment describing an official “read the generated dataplane credentials and push them to AWS Secrets Manager” pattern. Given that, the best we can say from the sources: - Yes, Airbyte (via bootloader) creates/manage dataplane credentials inside Kubernetes secrets, and in OSS Helm v1 it uses airbyte-auth-secrets as the place those keys are expected. [[1.6.0 chart issue](https://github.com/airbytehq/airbyte/issues/62468); [auth secret overwrite](https://github.com/airbytehq/airbyte/issues/64159)] - The knowledge sources do not document a supported method to: - programmatically retrieve those credentials from Airbyte in a stable way (apart from inspecting the K8s Secret directly), and then - sync them into an external secrets manager such as AWS Secrets Manager. So, within the limits of the docs/issues: - You can manually read whatever the bootloader has stored in the airbyte-auth-secrets (or the relevant dataplane secret in Enterprise Flex) and then put those values into AWS Secrets Manager yourself, but that is not described or guaranteed by the official docs. - There is no documented API or Helm option that says “export dataplane credentials to an external secrets manager” for OSS core. If you need an officially supported pattern for externalizing dataplane credentials, the existing documentation and issues do not provide one; you’d need guidance from the Airbyte team beyond what’s available in these sources.Eduardo Ferreira
11/24/2025, 10:04 PMkapa.ai
11/24/2025, 10:04 PMkapa.ai
11/24/2025, 10:05 PMairbyte-auth-secrets config?” 2. “Will my current secrets stay valid?” ### 1. Do you need to remove your airbyte-auth-secrets config? All the concrete troubleshooting we have for the dataplane-client-id error with Helm v1 shows it happening when airbyte-auth-secrets is pre‑created and managed externally (via values or an external secret manager). [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587); [dataplane missing key](https://github.com/airbytehq/airbyte/issues/62468)] In that 1.6.x case, the user resolved the exact same error by: - Commenting out the Helm values that override the auth secret, e.g.: yaml global: auth: instanceAdmin: secretName: airbyte-auth-secrets emailSecretKey: instance-admin-email passwordSecretKey: instance-admin-password - Deleting the existing airbyte-auth-secrets Secret, and - Letting Airbyte/bootloader recreate and populate it. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] After that, airbyte-auth-secrets contained the required dataplane-client-id / dataplane-client-secret, and workload‑launcher became healthy. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] So, based on the sources, the only clearly working pattern when you hit this error is: - Stop managing airbyte-auth-secrets via Helm values / external manager, and - Allow Airbyte to manage that secret itself. The docs and issues do not show a supported way to keep airbyte-auth-secrets fully external and satisfy the dataplane keys for OSS Helm v1. ### 2. Will your existing admin secrets stay valid? This is where the sources are incomplete: - In the 1.6.x workaround above, the user reports that after deleting their managed secret and redeploying, Airbyte created a new airbyte-auth-secrets and “**somehow it kept the same values for the instance admin keys**, but also created the dataplane-client-id and dataplane-client-secret.” [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] That’s observed behavior in that version, not a documented guarantee. - Another issue shows that the bootloader may overwrite values in the auth secret when doing dataplane initialization, which is why users asked for it not to rewrite credentials. [[auth secret overwrite](https://github.com/airbytehq/airbyte/issues/64159); [62468 comments](https://github.com/airbytehq/airbyte/issues/62468)] - The docs about authentication and airbyte-auth-secrets only describe how to: - Read the generated admin credentials, or - Override the admin password (and optionally client id/secret) by creating the secret yourself. [[Auth docs](https://docs.airbyte.com/platform/deploying-airbyte/integrations/authentication)] They do not state what happens if the platform later recreates or modifies that secret while also managing dataplane keys. Because of this, the knowledge sources: - Do not clearly guarantee that your current instance-admin-email / instance-admin-password (or other keys) will remain unchanged if you delete your managed secret and let Airbyte recreate it. - Only show one user’s report where they happened to remain the same after such a reset. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] So, from the available information: - To fix the dataplane error with Helm v1, the pattern supported by the issues is to remove your airbyte-auth-secrets overrides and delete the secret, letting Airbyte recreate it. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] - Whether your current admin credentials “stay valid” after that is not guaranteed by the docs; in at least one case they did, but another issue shows bootloader can overwrite auth secret content. [[1.6.0 abctl issue](https://github.com/airbytehq/airbyte/issues/58587); [auth secret overwrite](https://github.com/airbytehq/airbyte/issues/64159)] Given this, if you proceed with that workaround, you should plan as if your admin credentials might change and be ready to: - Read the new values from airbyte-auth-secrets after redeploy (via kubectl get secret ... as in the auth docs), and - Update any external references (e.g., your own Secret Manager) accordingly. [[Auth docs](https://docs.airbyte.com/platform/deploying-airbyte/integrations/authentication)]Eduardo Ferreira
11/24/2025, 10:06 PMkapa.ai
11/24/2025, 10:06 PMkapa.ai
11/24/2025, 10:07 PMdataplane-client-id / dataplane-client-secret problems are explicitly reported for Helm chart V1 and abctl installs around 1.6.x and 1.8.x, especially when airbyte-auth-secrets is pre‑created / externally managed. [[1.6.0 chart issue](https://github.com/airbytehq/airbyte/issues/62468); [1.6.x abctl issue](https://github.com/airbytehq/airbyte/issues/58587)] - The behavior of the bootloader around dataplane credentials was changed so that if valid credentials already exist in the secret, it will skip dataplane initialization (and thus stop constantly overwriting). This fix is available starting in release 1.8.0. [[bootloader fix](https://github.com/airbytehq/airbyte/issues/64159)] - Helm chart V2 is the new chart line recommended for 1.6.0+ Helm installs and is generally more consistent with best practices, but the V2 docs and implementation guides do not explicitly call out this dataplane/auth‑secret issue as “fixed by V2”. They mainly focus on changes in values structure, external DB/storage, and secret manager configuration. [[Chart V2 enterprise upgrade](https://docs.airbyte.com/platform/enterprise-setup/chart-v2-enterprise); [Next chart V2](https://docs.airbyte.com/platform/next/enterprise-setup/chart-v2-enterprise)] From this, the safest statement based on the sources is: - Moving to a newer Airbyte version (≥1.8.0) that contains the bootloader fix should reduce problems with the auth secret constantly being rewritten for dataplane credentials, regardless of chart version. [[bootloader fix](https://github.com/airbytehq/airbyte/issues/64159)] - Using Helm chart V2 is recommended for new/updated Helm deployments and may avoid some of the chart‑level rough edges seen in V1, but the docs and issues provided do not explicitly guarantee that simply switching to V2 will, by itself, eliminate all dataplane-client-id missing/secret‑management conflicts. So: upgrading to a version that includes the bootloader change (1.8.0+) and using chart V2 is likely to behave better than older V1 setups, but the knowledge sources do not explicitly state “Helm V2 fixes this issue,” so I cannot confirm that as a documented fact.