Can I Integrate a secrets manager in my Airbyte instance to store my connector credentials?

Yes, you can integrate a secrets manager in your Airbyte instance to store your connector credentials. If you're deploying Airbyte Open Source on Google Cloud Platform (GCP), you can use Google Secret Manager to store credentials instead of in the database. Here are the steps to do this: 1. Create a service account with Google Secret Manager with read/write access. Generate a JSON key. 2. In the Worker and Server applications, set the

SECRET_STORE_GCP_PROJECT_ID

environment variable to the GCP project to which the credentials have access and secrets will be located. 3. In the Worker and Server applications, set the

SECRET_STORE_GCP_CREDENTIALS

environment variable to the JSON key created in step 1. 4. In the Worker and Server applications, set the

SECRET_PERSISTENCE

environment variable to

GOOGLE_SECRET_MANAGER

. Please note that this process is not reversible. Once you have converted to a secret store, you won’t be able to reverse it. For more details, you can refer to the [Airbyte Security documentation](https://docs.airbyte.com/operator-guides/security).

@kapa.ai I plan to use the Bitwarden secrets manager cli to store the credentials using Bitwarden. To use the cli I would need to run the commands using docker. How do I do this with Airbyte? Which docker file should I be modifying so I can use the connector credentials? Do I need to modify each individual connector docker file?

Hmm, I don't know enough to give you a confident answer yet. The provided extracts do not contain specific information about using Bitwarden secrets manager CLI with Airbyte. However, they do provide some general guidance on how to use Docker with Airbyte and how to manage credentials. In general, to use Docker with Airbyte, you would need to build the Docker image for your connector and then run the necessary commands. For example, to build the Docker image, you would use the command

docker build . -t airbyte/destination-<name>:dev

or

docker build . -t airbyte/source-python-http-example:dev

depending on whether you're building a source or destination connector ([source](https://docs.airbyte.com/connector-development/tutorials/building-a-python-destination), [source](https://docs.airbyte.com/connector-development/tutorials/cdk-tutorial-python-http/use-connector-in-airbyte)). As for managing credentials, Airbyte provides a few options. You can use Google Secret Manager to store credentials ([source](https://docs.airbyte.com/operator-guides/security)), or you can use the secret management feature to avoid storing credentials in the YAML files ([source](https://docs.airbyte.com/cli-documentation)). You can also mount the inputs into the Docker container to allow the connector to access your secrets without having to build them into the container ([source](https://docs.airbyte.com/connector-development/tutorials/build-a-connector-the-hard-way)). However, these extracts do not provide specific information on how to use Bitwarden secrets manager CLI with Airbyte or which Docker file you should be modifying. For more detailed guidance, you may want to refer to the [Airbyte documentation](https://docs.airbyte.com/) or ask for help in the [Airbyte community](https://airbytehq.slack.com).

@kapa.ai how do I modify my connector docker file to include the following example changes  RUN SECRET_1=$(bws secret get fc3a93f4-2a16-445b-b0c4-aeaf0102f0ff --access-token $BWS_ACCESS_TOKEN | jq '.value')  RUN SECRET_2=$(bws secret get 80b55c29-5cc8-42eb-a898-acfd01232bbb --access-token $BWS_ACCESS_TOKEN | jq '.value') These RUN statements will prompt your Dockerfile to retrieve the indicated secrets, where fc3a93f4-2a16-445b-b0c4-aeaf0102f0ff represents the secret's specific identifier. Prepare your env file Now that your database credentials will be made available for injection, condition your settings.env file to be able to receive these values. To do this, replace relevant hardcoded values in the file with the designated variable names (in this case, SECRET_1 and SECRET_2):  # Database # Available providers are sqlserver, postgresql, mysql/mariadb, or sqlite BW_DB_PROVIDER=mysql BW_DB_SERVER=db BW_DB_DATABASE=bitwarden_vault BW_DB_USERNAME=$SECRET_1 BW_DB_PASSWORD=$SECRET_2 Run the container Now that your database credentials are primed and ready for injection, start up your container and specify the access token to use with bws login as an environment variable:  docker run -e BWS_ACCESS_TOKEN=<your-access-token> docker-unified When this command is run, your Docker container will start up and inject your database credentials from the values retrieved by the container, allowing you to securely spin up Bitwarden Unified without passing sensitive values as plaintext.

A message from kapa.ai