(Note originally posted this in the connections help channel by mistake. I think this is the right place) After following the Helm/Kubernetes guide on the Airbyte site for a Google Kubernetes Engine install I have been unable to get Airbyte to use a database secret I stored in Kubernetes. After setting

externalDatabase.existingSecret

and

externalDatabase.existingSecretPasswordKey

I still get this error on bootloader startup. Seems like the bootloader is not respecting the external database settings, it's not looking at my custom Kubernetes secret and defaulting to

airbyte-airbyte-secrets

.

couldn't find key DATABASE_PASSWORD in Secret default/airbyte-airbyte-secrets: CreateContainerConfigError

If I redo it with the actual password into values.yaml it works, but I don't want to check it in to source control like that. I believe my issue is the same one described here (https://airbytehq.slack.com/archives/C021JANJ6TY/p1681755909155819) but I didn't see a resolution. I am using chart version

airbyte-0.48.10

Based on the information provided in the extracts, it seems like you're trying to use an external database with Airbyte on Kubernetes and you're facing issues with the database secrets. In the

values.yaml

file, you should provide your external DB secrets under the

global.database

section in the following format:

yaml database:     secretName: "myOctaviaSecret"     secretValue: "postgresql-password"     host: "<http://example.com|example.com>"     port: "5432"

Then, you should upgrade the chart by running:

shell helm upgrade -f path/to/values.yaml %release_name% airbyte/airbyte

This information is based on the extract from the [Airbyte documentation](https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm). However, if you're still facing issues, it might be due to a bug or a configuration issue that's not covered in the provided extracts. In such cases, it would be best to reach out to the Airbyte community on their [Slack channel](https://airbytehq.slack.com/archives/C021JANJ6TY/p1665150128170829) or [forum](https://discuss.airbyte.io/t/2928) for more specific assistance.

I did try that, there are two separate config pairs to set DB secrets

global.database.(secretName|secretValue)

and

externalDatabase.(existingSecret|existingSecretKey)

I've tried each of those and both, and in all cases the airbyte-bootloader still gives me

couldn't find key DATABASE_PASSWORD in Secret default/airbyte-airbyte-secrets: CreateContainerConfigError

which causes the helm install command to timeout.

If it helps, this worked for me with a GKE installation... First, I create my K8s secret using:

kubectl create secret generic airbyte-db-creds --from-literal=DATABASE_USER=airbyte --from-literal=DATABASE_PASSWORD=[insert password]

Set the following in the

global

section of the values.yaml file:

database:
    secretName: "airbyte-db-creds"
    secretValue: "DATABASE_PASSWORD"
    host: "airbyte-db-proxy"
    port: "5432"

Note I'm using a Cloud SQL proxy to connect to my database, with a K8s service named airbyte-db-proxy. In the

externalDatabase

section, I have:

externalDatabase:
  host: "airbyte-db-proxy"
  user: airbyte
  existingSecret: "airbyte-db-creds"
  existingSecretPasswordKey: "DATABASE_PASSWORD"
  database: airbyte-db
  port: 5432
  jdbcUrl: "jdbc:<postgresql://airbyte-db-proxy:5432/airbyte-db>"

Hope this helps!

Hey @Jamie Geddes your comment helped me quite a bit with getting the credentials to work at least (I think), but I'm getting an error with the connection to my Cloud SQL instance. I'm trying to connect directly to it with the Private IP of the instance (both Cloud SQL and the Airbyte instance are deployed on the same GCP network). You've used the

airbyte-db-proxy

and I was wondering if you could offer some additional information about that? Networking isn't my forte so I'm making it up as I go!

Hey @Danilo Drobac, I'm definitely no networking expert either, so a lot of this was trial and error 🙂 I'm using the Cloud SQL proxy from Google, rather than connecting to the private IP of the database. My PostgreSQL Cloud SQL instance is set up to use IAM authentication. This means I need to enable workload identity bindings between the IAM service account I'm using, and the K8s service account that Airbyte uses. First of all, I setup the variables and connect to my GKE cluster:

export ENV=staging
export PROJECT_ID=airbyte-infra-$ENV
export REGION=europe-west1
export CLUSTER=airbyte-cluster
export K8S_SA=airbyte-admin
export IAM_SA=airbyte-sa
export NAMESPACE=default

gcloud config set project $PROJECT_ID
gcloud container clusters --region $REGION get-credentials $CLUSTER --project=$PROJECT_ID

I then create a K8s service account (with the name set to use the variable defined above):

kubectl create serviceaccount ${K8S_SA}

I then define the workload identity binding:

gcloud iam service-accounts add-iam-policy-binding \
--role="roles/iam.workloadIdentityUser" \
--member="serviceAccount:${PROJECT_ID}.svc.id.goog[${NAMESPACE}/${K8S_SA}]" \
${IAM_SA}@${PROJECT_ID}.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>

and add an annotation:

kubectl annotate serviceaccount \
${K8S_SA} \
<http://iam.gke.io/gcp-service-account=${IAM_SA}@${PROJECT_ID}.iam.gserviceaccount.com|iam.gke.io/gcp-service-account=${IAM_SA}@${PROJECT_ID}.iam.gserviceaccount.com>

I then setup a

clusterrolebinding

to allow the K8s service account to create pods within the GKE cluster - this is required for Airbyte to be able to actually run a sync operation, and can lead to confusing error messages if not set:

kubectl create clusterrolebinding default-airbyte-admin \
    --clusterrole cluster-admin \
    --serviceaccount=${NAMESPACE}:${K8S_SA} \
    --namespace ${NAMESPACE}

Note the name for the binding here is arbitrary, the important thing is to setup the

cluster-admin

role for the K8s service account. I then deploy my Cloud SQL proxy yaml file (see attached file, note my Cloud SQL instance is named

airbyte-master

) via the following:

kubectl apply -f sqlproxy-deployment.yaml

The remaining steps then start with the commands in my previous post (i.e. from creating the K8s secret onwards). You'll need to ensure you edit the values.yaml file for your Airbyte deployment to ensure it doesn't try to create the K8s service account, as you want to use the one you created earlier with the associated workload identity bindings, to allow it to use the GCP IAM service account to connect to your Cloud SQL instance:

serviceAccount:
  create: false

Hope this helps 👍

You're a legend, thank you! I will go through this in more detail later but just having something to base it off of is really great 😃

One minor correction, the

sqlproxy-deployment.yaml

file contains a

$PROJECT_ID

reference, so you can substitute this when you deploy using:

cat sqlproxy-deployment.yaml | sed "s/{{PROJECT_ID}}/$PROJECT_ID/g" | kubectl apply -f -