https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> - How can we set environment variab...
# replication-ask-ai
s
@kapa.ai - How can we set environment variables from external secrets ?
k
Airbyte supports using external secret managers for connector secrets, but the docs and issues you shared do not describe a general mechanism to automatically expose those external secrets as environment variables for connectors or core services. From the available information: - For connector credentials, the recommended pattern is to configure an external secrets manager (AWS Secrets Manager, Google Secret Manager, Azure Key Vault, Hashicorp Vault). Airbyte then reads/writes connector secrets there instead of storing them in its own database or Kubernetes secrets. [[Secret mgmt](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets); [External connector secrets](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#configuring-external-connector-secret-management)] - On Kubernetes, you still define Kubernetes 
Secret
objects that hold the credentials to access those external managers (e.g. AWS access key, GCP service account JSON, Azure client secret). These are referenced in 
values.yaml
under 
global.secretsManager
(Helm v2) or 
secretsManager
(Helm v1). [[External connector secrets](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#configuring-external-connector-secret-management)] - The examples for environment variables in the Helm chart show how to map values from Kubernetes Secrets or ConfigMaps into pod env vars via 
extraEnv
, not from external secret managers directly. [[Helm env example](https://github.com/airbytehq/airbyte/issues/49843); [Temporal certs secret](https://github.com/airbytehq/airbyte/issues/69089)] There is no documented feature in these sources that: - pulls values from AWS/GCP/Azure secret managers and - automatically injects them as environment variables into Airbyte pods. Instead, the documented patterns are: 1. Use external secret managers for connector configs (Airbyte handles reading/writing secrets for connectors internally). [[Secret mgmt](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets)] 2. Use Kubernetes Secrets → env vars for core service configuration (e.g. DB passwords, Temporal certs) by: - creating a 
Secret
in your cluster, then - referencing it via 
secretKeyRef
in 
extraEnv
or in your own manifests. [[Helm env example](https://github.com/airbytehq/airbyte/issues/49843); [Temporal certs secret](https://github.com/airbytehq/airbyte/issues/69089)] Because the knowledge sources don’t show a direct “external secret manager → env var” integration, the safest statement is: > To set environment variables, you must expose values via Kubernetes Secrets (or ConfigMaps) and reference them as env vars in the pod spec / Helm 
extraEnv
. External secret managers are supported for connector secrets, but not documented as a generic env-var source. If you share your deployment method (Helm chart version, Kubernetes vs Docker, etc.), I can point to the exact 
values.yaml
fields that are shown in the docs for wiring a Kubernetes Secret into env vars.
s
@kapa.ai How to configure extraEnv using external secrets
k
A message from kapa.ai
The knowledge sources you provided do not document a way to wire external secret managers directly into 
extraEnv
. They only show: 1. How to configure external secret managers for connector secrets (AWS Secrets Manager, Google Secret Manager, Azure Key Vault, Hashicorp Vault). [[Secret mgmt](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets)] 2. How to configure Kubernetes Secrets that hold the credentials to access those external managers (e.g. 
airbyte-config-secrets
with AWS access key / secret key, GCP 
gcp.json
, Azure client secret). [[Secrets](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets); [Values](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets#values)] From these docs, the supported pattern is: - External secret manager ⟶ used internally by Airbyte for connector configuration secrets (not as env vars). - Kubernetes 
Secret
⟶ referenced in Helm 
values.yaml
under 
global.secretsManager.secretName
to provide credentials for the external manager. [[Values](https://docs.airbyte.com/platform/deploying-airbyte/integrations/secrets#values)] - 
extraEnv
(when used) expects values or 
secretKeyRef
from Kubernetes Secrets, not directly from AWS/GCP/Azure secret managers. The provided sources do not show any example of 
extraEnv
pulling from external secret managers. So, based on the available documentation: > To configure 
extraEnv
, you must first expose values via a Kubernetes Secret (or ConfigMap) and then reference that secret in 
extraEnv
. The docs do not describe a mechanism to bind 
extraEnv
directly to AWS Secrets Manager / Google Secret Manager / Azure Key Vault / Hashicorp Vault. If you need a concrete 
extraEnv
example, it would have to be of the form “Kubernetes Secret → env var”; the knowledge sources you shared don’t include such a snippet, only the secret-manager configuration itself.
s
@kapa.ai - How can I pass additional environment variables to pods that run replication jobs ?
k
A message from kapa.ai
You can pass additional environment variables to the pods that run replication jobs by using the 
extraEnv
 mechanism on the relevant components in your Helm 
values.yaml
. However, the knowledge sources only show 
extraEnv
usage for core services (e.g., 
server
, 
temporal
, 
airbyte-bootloader
) and do not explicitly document a dedicated 
extraEnv
block that targets only replication job pods. What is documented: - Helm chart v2 introduces many job-related settings under 
global.jobs.kube.*
(node selectors, tolerations, images, etc.), but not arbitrary env vars for job pods. [[Chart v2 mapping](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file)] - For other pods (e.g., Temporal, bootloader), extra env vars are added via 
extraEnv
in their section of 
values.yaml
, using standard Kubernetes 
value
/ 
valueFrom.secretKeyRef
/ 
configMapKeyRef
. [[Temporal extraEnv example](https://github.com/airbytehq/airbyte/issues/48879); [Bootloader extraEnv workaround](https://github.com/airbytehq/airbyte/issues/65154)] - The upgrade docs show how job pod placement is controlled via 
global.jobs.kube.*
instead of env vars like 
JOB_KUBE_NODE_SELECTORS
, 
SPEC_JOB_KUBE_NODE_SELECTORS
, etc. [[Chart v2 jobs mapping](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file)] Because there is no explicit example in the provided sources of “replication job pods `extraEnv`”, the safest, documented options are: 1. Use the dedicated Helm fields for job behavior For anything covered by chart v2 (node selectors, tolerations, images, resources), configure:
Copy code
yaml    global:      jobs:        kube:          nodeSelector:            type: jobs          scheduling:            check:              nodeSelectors:                type: jobs            discover:              nodeSelectors:                type: jobs            spec:              nodeSelectors:                type: jobs
This replaces older env vars like 
JOB_KUBE_NODE_SELECTORS
, 
CHECK_JOB_KUBE_NODE_SELECTORS
, etc. [[Chart v2 jobs mapping](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file)] 2. Use 
extraEnv
on core services when you need env vars there For example, to add env vars to the Temporal pod you can do:
Copy code
yaml    temporal:      extraEnv:        - name: DYNAMIC_CONFIG_FILE_PATH          value: config/dynamicconfig-override/development.yaml
[[Temporal extraEnv example](https://github.com/airbytehq/airbyte/issues/48879)] Given the docs you shared, there is no clearly documented knob like 
global.jobs.extraEnv
or similar that injects arbitrary env vars directly into replication job pods. If you need that level of control, it is not described in these sources, so I cannot point to a specific, supported 
values.yaml
key for it.
6 Views
Open in Slack
PreviousNext
Powered by