@kapa.ai Context handoff: Airbyte v2 (Helm chart 2.1.0) on self-hosted k3s — AWS auth fails in workload pods The core need: make AWS authentication (to AWS Secrets Manager, and S3) succeed from Airbyte's dynamically-launched workload/discover pods, which run on self-hosted k3s — NOT on AWS. So instance-profile / IMDS is not an option. Static credentials are completely fine — whatever makes auth work. (IRSA/OIDC via web identity was explored as one lead but is not a requirement; don't anchor on it.) ## Stack - Pulumi-managed Airbyte, Helm chart

airbyte-v2

(repo https://airbytehq.github.io/charts), version 2.1.0, Airbyte 2.1.0, edition COMMUNITY. - Self-hosted k3s via k8s3s (NOT EKS), so pods are not on AWS and have no IMDS / instance profile ## Root cause (confirmed by reading the chart templates in the pulled chart

/tmp/airbyte_chart/airbyte/templates/config/

) - The discover init container hydrates connector secrets, so it must authenticate to AWS Secrets Manager itself. - The workload-launcher forwards only ConfigMap-backed env (

-airbyte-env

) into the dynamically-launched workload/discover pods. Every

secretKeyRef

-backed env is dropped. - In the chart, AWS credentials — both

storage.s3

(`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`) and

secretsManager.awsSecretManager

(`AWS_SECRET_MANAGER_ACCESS_KEY_ID`/`_SECRET_ACCESS_KEY`) — are emitted as

secretKeyRef

(see

_storage.tpl:180

,

_secretsManager.tpl:54/79

). Only region/tags/kms/persistence-type go into the ConfigMap (

_secretsManager.tpl:562

configVars

). - Result: discover init pod env has

SECRET_PERSISTENCE=AWS_SECRET_MANAGER

,

AWS_SECRET_MANAGER_REGION

, tags, kms — but no credentials at all. The AWS SDK falls through its default chain (env → IMDS) and dies:

Unable to load AWS credentials from any provider in the chain

. (IMDS unreachable — expected, not on AWS.) - The chart's Azure path emits `AB_AZURE_KEY_CLIENT_ID_REF_NAME`/`_REF_KEY` into the ConfigMap so workload pods can self-reference the secret; the **AWS path has no such `*_REF_NAME`/`*_REF_KEY`** — looks like a chart gap for AWS Secret Manager +

credentials

auth + remote workloads. (

global.workloads.secretName

in values is referenced nowhere in templates — dead value.) ## What we already fixed (correct, keep these) 1. `secretsManager`: added

enabled: true

, switched to

authenticationType: credentials

, removed 1.x `accessKeyIdSecretKey`/`secretAccessKeySecretKey` (those don't exist in v2 secretsManager). 2.

airbyte-config-secrets

keys for secret manager must be uppercase

AWS_SECRET_MANAGER_ACCESS_KEY_ID

/

AWS_SECRET_MANAGER_SECRET_ACCESS_KEY

(chart default keys). Platform pods (server/worker/launcher) now run fine. Storage block is correct as-is (

accessKeyIdSecretKey

IS valid for storage,

_storage.tpl:173

). The platform side works; only the launched workload/discover pods can't get AWS creds. Old cluster (chart 1.8.5) worked because secret hydration happened in the worker (a platform pod that has the creds), not in a per-job init container — this moved in v2. ## The open problem to solve in the new session Make the launched discover/check/sync workload pods carry working AWS credentials so the init container can hit Secrets Manager. The launcher only forwards the

-airbyte-env

ConfigMap (non-secret) into these pods, so the static keys never arrive. Find the Airbyte-supported way to inject credentials into connector/workload pods, e.g.: - A chart/Airbyte setting that puts the AWS creds (or a `*_REF_NAME`/`*_REF_KEY` pointer) into the connector-pod env or mounts the

airbyte-config-secrets

secret into workload pods. - Airbyte support for extra env / extra volumes / volume mounts / envFrom on connector & workload (job) pods (look for

JOB_KUBE_*

,

global.workloads

, container-orchestrator settings, connector-pod annotations, pod-template/overrides, `extraVolumes`/`extraVolumeMounts` for jobs). - Worst case acceptable: get `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` into the forwarded ConfigMap so the SDK default chain finds them (plaintext-in-configmap tradeoff — note it but it's an option since these are the same S3 IAM keys). ## Please research (Airbyte docs + web), don't just reason from the chart - Airbyte v2 / Helm chart 2.x official docs on external secret managers with Helm/abctl on non-EKS Kubernetes, and how credentials reach connector/workload pods. Start at https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets plus the storage and workloads pages. - How Airbyte injects extra volumes / mounts / env into connector & workload pods (

JOB_KUBE_*

,

global.workloads

, container-orchestrator, pod overrides). - Known GitHub issues: "AWS Secret Manager credentials not passed to workload/init pods" / "discover pod Unable to load AWS credentials" on Helm chart v2 / Airbyte 1.7+/2.x — there may be a known fix, required value, or chart version bump.

AWS Credentials in Workload/Discover Pods — Helm Chart v2 on k3s The knowledge sources confirm your root-cause analysis and shed light on the known issues, but do not contain a fully documented, officially supported fix for the exact scenario (non-EKS k3s + AWS Secret Manager credentials auth + workload/discover pods). Here is what the sources do tell us: 1. Confirmed: This is a Known Bug The issue you've hit is a real, open regression in the v2 workload-launcher. Two GitHub issues confirm it: Issue #69233 (Azure, but same root cause) identifies the bug precisely in `EnvVarConfigBeanFactory.kt`: [Init container bug] >

checkEnvVars

and

discoverEnvVars

beans only inject

airbyteMetadataEnvMap

into check/discover pods. They're missing

secretPersistenceEnvMap

(plain env vars) and

secretPersistenceSecretsEnvMap

(Kubernetes secret refs). > The

syncEnvVars

bean (used for sync pods) correctly includes all three maps, which is why sync works. Only check/discover pods are missing them. A community contributor posted the exact fix needed in

EnvVarConfigBeanFactory.kt

— but since the platform repo doesn't accept PRs, it's not yet merged. Issue #67588 (AWS Secret Manager, Helm chart 2.0.17) is essentially your exact problem — credentials configured correctly in

airbyte-config-secrets

, platform pods work, but connector/workload pods fail with

Unable to load AWS credentials from any provider in the chain

. [AWS secret manager error] 2. The Workaround That Has Worked (from Issue #48809) A user on a similar issue (Airbyte 1.2.0, EKS) found that adding the following to

values.yaml

under

workload-launcher

(v1) /

workloadLauncher

(v2) caused the spawned pods to receive the secret refs: [workload-launcher workaround]

workloadLauncher:
  env_vars:
    AWS_ASSUME_ROLE_SECRET_NAME: airbyte-config-secrets

    AWS_SECRET_MANAGER_ACCESS_KEY_ID_REF_NAME: airbyte-config-secrets
    AWS_SECRET_MANAGER_ACCESS_KEY_ID_REF_KEY: AWS_SECRET_MANAGER_ACCESS_KEY_ID

    AWS_SECRET_MANAGER_SECRET_ACCESS_KEY_REF_NAME: airbyte-config-secrets
    AWS_SECRET_MANAGER_SECRET_ACCESS_KEY_REF_KEY: AWS_SECRET_MANAGER_SECRET_ACCESS_KEY

> Note: This workaround was confirmed on chart v1 (1.2.0). Whether the `*_REF_NAME`/`*_REF_KEY` pattern is still wired up in the v2 workload-launcher's pod-building logic is not confirmed by the knowledge sources. Given your analysis that the AWS path lacks this in v2 templates (unlike Azure), it may not work — but it is worth trying first since it's the closest documented workaround. 3. The "Plaintext in ConfigMap" Escape Hatch Your own analysis identified this: if the `*_REF_NAME`/`*_REF_KEY` approach doesn't work, putting

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

directly into

global.env_vars

(which lands in the

-airbyte-env

ConfigMap that IS forwarded to workload pods) would make the AWS SDK default chain find them. The knowledge sources do not explicitly endorse or document this approach, but it is consistent with how the ConfigMap forwarding works as you've described it.

global:
  env_vars:
    AWS_ACCESS_KEY_ID: "your-key-id"          # plaintext tradeoff — note security implications
    AWS_SECRET_ACCESS_KEY: "your-secret-key"  # same

This is a security tradeoff (credentials in a ConfigMap, not a Secret), but it is a pragmatic option for a self-hosted, non-production or air-gapped environment. 4. What the Official Docs Say The official secret management docs describe the

values.yaml

structure for

global.secretsManager

with

authenticationType: credentials

, and state that credentials should be in the

airbyte-config-secrets

Kubernetes secret under

aws-secret-manager-access-key-id

/

aws-secret-manager-secret-access-key

keys. [Secret Management docs] However, the docs do not address how those credentials reach dynamically-launched workload/discover pods — this gap is exactly what the open GitHub issues are about. Summary of Options (in order of preference) The most actionable next step is to try the

workloadLauncher.env_vars

workaround from issue #48809 first. If the `*_REF_NAME`/`*_REF_KEY` variables are still consumed by the v2 launcher's pod-builder, it will work without exposing credentials in plaintext. If not, the plaintext ConfigMap approach is your fallback until the upstream bug is fixed. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements