Hello all, quick question when storing secrets via the UI I can see the are saved into the associated db (for me its postgres) as plain text is this intentional and if so, is there a more secure method of managing secrets with AWS Secrets Manager / Azure Key vault or others?

Today Airbyte doesn’t support external secret store 😞 only the option with the airbyte-db. There are Github issues to improve the security of secrets in the associated database, but currently this is the state of this feature.

Thanks Marcos that’s really helpful mate

I guess there has been some progress made on API creds storage. Is this extensible to AWS secrets manager?

both AWS Secrets Manager and Google Secrets Manager (and maybe Hashicorp Vault?) are supported for secrets management. This is clearest examples of this are in the Self-Managed Enterprise (SME) deployment docs, but also apply to OSS: https://docs.airbyte.com/enterprise-setup/implementation-guide#configuring-external-connector-secret-management We're currently using Google Secrets Manager on GCP and it's working great—just note that if you're moving an existing deployment it doesn't automatically migrate the secrets, so you need to script that. (You'll find some scripts I wrote for both AWS and GCP to semi-automate that on this GitHub issue.)

thanks @Justin Beasley for the detailed response. Looks very promising. In your case if you have a new connector that requires a secret/key for external source e.g. REST api, you would update the your

gcp.json

and re-build?

@Christopher Greene No, the

gcp.json

key just holds the encoded service key JSON used to authenticate to Secret Manager—every credential used in a source or destination is stored as its own secret in Secrets Manager. So the only difference is that instead of storing them in the

secrets

table in Airbyte's internal Postgres database (in plaintext), they're instead served encrypted in the Secrets Manager. This means that even if someone got your Airbyte DB (or a backup thereof), they wouldn't have the connection secrets. And if you were ever concerned about an access issue of any type, you could revoke the secrets manager access for that service account and it could no longer read the secrets.

ok got it! thanks for the explanation