Hello Airbyte team we re testing your product in Kubernetes Airbyte #feedback-and-requests

Hello Airbyte team, we're testing your product in ...

Lukas Novotny

01/21/2022, 11:24 AM

Hello Airbyte team, we're testing your product in Kubernetes and it's working great! However, we are concerned about the unencrypted connection config storage in Postgres

actor

table. Are we missing some trivial config that switches to a different secret store? Can we load secrets from environment variables?

Jens

01/21/2022, 1:29 PM

Hi @Lukas Novotny if you're on GCP you can use GSM to store secrets, this can be set through env vars, check this doc and let me know if it helps.

Jens

01/21/2022, 1:32 PM

Unfortunately we're not on GCP, do we have other options?

Anouar Hnini

01/21/2022, 3:45 PM

Not at this time. We have a long standing issue to address this https://github.com/airbytehq/airbyte/issues/837

Anouar Hnini

01/23/2022, 3:41 PM

Hey @[DEPRECATED] Augustin Lafanechere - I enabled GSM by adding an entry in the the .env file at

/kube/overlays/stable/.env

like so

Copy code

SECRET_PERSISTENCE_TYPE=GOOGLE_SECRET_MANAGER

And I verified by looking at the ConfigMap in GKE that the value is set. However no secrets seem to be created in my GSM and the server logs are still saying

Copy code

2022-01-23 15:35:52 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-01-23 15:35:52 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-01-23 15:35:52 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):596 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'

So looks like it’s not picking up the env var correctly? Or do I need set all of the variables defined in the doc you linked? I currently just set the

SECRET_PERSISTENCE_TYPE

Edit: I tried setting all of them but still getting the same result

Anouar Hnini

01/23/2022, 4:16 PM

In fact seems that no matter what I put in the

.env

file, the server logs will still show Airbyte choosing the default value (even when the ConfigMap reflects the value in the .env

Anouar Hnini

01/23/2022, 6:45 PM

Btw found that above is due to these env vars not being defined in the kubernetes manifests. I added them there and now I can override the defaults.

Yashasvi Chaudhary

01/24/2022, 4:00 PM

That said though… still seems theres issues getting the GCP service account credentials to format correctly… Do you have an example of how to configure e.g. the

SECRET_STORE_GCP_CREDENTIALS

in the .secrets kustomize env file? @[DEPRECATED] Augustin Lafanechere

Yashasvi Chaudhary

01/24/2022, 4:13 PM

From the looks of it…. it seems that the secrets are stored in GSM as json objects, but then Airbyte fails to parse them as it expects a string. So this might be a bug tbh. Recreate: 1. Use GSM 2. Try to set up new BigQuery source pasting JSON credentials in the Airbyte UI 3. These credentials are stored in GSM as JSON objects 4. Airbyte tries to read them when checking connection with the source (and fails to parse them as it expects a string)

Yashasvi Chaudhary

01/24/2022, 4:43 PM

I think there is a bug 🐛 . The documentation at https://docs.airbyte.com/operator-guides/configuring-airbyte#secrets instruct to set a secret

SECRET_PERSISTENCE_TYPE

however, the code expects

SECRET_PERSISTENCE

https://github.com/airbytehq/airbyte/blob/5a091b12460ec6773c5655e3cae63248ad5cea86[…]e-config/models/src/main/java/io/airbyte/config/EnvConfigs.java

Yashasvi Chaudhary

01/24/2022, 5:30 PM

Yeah that is also a bug. Forgot to mention that 😅 But that I already worked around by setting

SECRET_PERSISTENCE

instead. So even working around that bug, the above still happens which I think is a separate bug

Eugene Krall

01/25/2022, 10:02 AM

@[DEPRECATED] Augustin Lafanechere See above for 1 (possibly 2) bugs regarding the google secret manager implementation

Elias Djurfeldt

01/26/2022, 9:58 PM

Hey @Elias Djurfeldt thank you very much for this investigation! I'll open the issue you mention on our repo tomorrow, feel free to open them yourself.

Elias Djurfeldt

01/26/2022, 10:01 PM

Just to summarize, here are the three problems you raised: • GSM related env vars are not declared in Kube manifests • SECRET_PERSISTENCE_TYPE in documentation should be SECRET_PERSISTENCE • Deserialization of GSM secrets fails during connection check

Elias Djurfeldt

01/26/2022, 10:33 PM

The third issue is also a problem of documentation (specifying the format for the secrets stored in the env variable)

Jens

01/27/2022, 9:34 AM

Great - thank you @[DEPRECATED] Augustin Lafanechere! Yes as @Jared Rhizor (Airbyte) says, the third issue would also benefit from some additional documentation on expected format of the secrets. I tried a bunch of different formats and ways of specifying the service account before realising that there’s probably a bug in the deserialization.

Daniele

01/27/2022, 6:23 PM

@Elias Djurfeldt, I created two new issues: • 🐛 Secrets store: secrets related env vars are not declared in Kubernetes manifests • 🐛 Secret store: Deserialization of GSM secrets fails during connection check : Please share the Airbyte version you are using on this one Feel free to subscribe to these to receive updates from our team, and share the details of your investigations if needed. I opened a PR to fix the env var naming in the documentation.

Ping-Lin Chang

01/28/2022, 10:08 AM

Awesome - thank you! The bugs were found in

0.35.6-alpha

Open in Slack

Previous Next