Hey all, trying to evaluate whether this is a valid configuration for GCP: • Airbyte running on GKE • Application secrets stored in GCP Secret Manager • Use Workload Identity & client libraries on GKE to access • Load into job configuration when running Airbyte Has anyone accomplished something similar before? Were there any hard roadblocks when trying this implementation?

I’m going to tag @Davin Chia (Airbyte), one of our engineers who works in the operations area, to try to get back to you on this. We’ve done something similar for airbyte cloud, but I don’t know enough of the fine details to respond myself. He’s in a time zone a ways off, though - expect he won’t see this until tomorrow.

Thanks so much! Very much appreciate any help we can get during our eval period.

Hi! This got lost in my messages. The first 3 points is basically the same as what we do on Cloud. I’m not too sure what your last point about

load into job configuration

means. Can you say more?

Hi @Davin Chia (Airbyte) awesome that’s great to hear! Yes to clarify - step 4 in this scenario is to use the key accessed to perform the integration run. Use case is to be hitting 4 different instances of an app on rotating keys. Bonus would be to add a metadat parameter on load to tag the source system (but I haven’t gotten that far on POC).

Yeap that makes sense to me. @Jenny Brown we have native support for GSM in OSS yes? So this should be as easy as setting some env vars

Oh Jenny is out this week. @Jared Rhizor (Airbyte) do you know?

We do have the ability to store secrets for a new GSM store, but the current model makes Airbyte responsible for selecting the name of a secret and actually creating it. We don’t yet have the ability to plug in to existing secrets and say “use existing GSM secret id xyz for this field in a config”. It’s more of a storage mechanism at this point.

Possible workaround may be to prebuild the yaml configs and mount to the cluster prior to job run?

Assuming use of Airflow the config build would just be part of the DAG