Does anyone have a best practice regarding backing...
# ask-ai
k
Does anyone have a best practice regarding backing up and restoring credentials? For instance, we’d like to be able to do the following: • backup: regularly back up sources, destinations & connections in a git repo • restore: in the event that our airbyte installation gets messed up somehow, be able to restore all the sources, destinations & connections from the yaml files The backup step can be done pretty easily by exporting these things to yaml files and saving the yaml files in the repo. But the restore step is a little more tedious / painful because all the credentials would have to be added by hand, locally. And I don’t know if I want to write a script that somehow pulls all the credentials from <somewhere> because just having a script like that could be a security vulnerability?
k
A message from kapa.ai
You can use the Octavia CLI to manage your Airbyte resource configurations, including sources, destinations, and connections, and store them in a Git repository for version control. This will help you with the backup process. However, for the restore process, you'll need to handle the credentials carefully to avoid security vulnerabilities. To securely manage credentials, you can use environment variables and store them in your
~/.octavia
file. Then, reference these variables in your configuration files using the
${VARIABLE_NAME}
syntax. This way, you won't store plain-text credentials in your YAML files, and you can restore configurations without exposing sensitive information. Here's an example of how to reference environment variables in a source configuration file:
Copy code
yaml cat sources/postgres/configuration.yaml  resource_name: "postgres" definition_type: source definition_id: decd338e-5647-4c0b-adf4-da0e75f5a750 definition_image: airbyte/source-postgres definition_version: 1.0.14  configuration:   host: ${POSTGRES_HOST}   port: 2000   schemas: ["public"]   database: ${POSTGRES_DATABASE}   password: ${POSTGRES_PASSWORD}   ssl_mode:     mode: "disable"   username: ${POSTGRES_USERNAME}   tunnel_method:     tunnel_method: "NO_TUNNEL"   replication_method:     method: "Standard"
Source: [Version control Airbyte configurations with Octavia CLI](https://airbyte.com/tutorials/version-control-airbyte-configurations) Keep in mind that Octavia CLI manages Airbyte Configurations as Code (CaC) and not Airbyte Infrastructure as Code (IaC). Thus, Octavia CLI cannot provision an Airbyte instance. For more information on using Octavia CLI, refer to the [tutorial on version control Airbyte configurations](https://airbyte.com/tutorials/version-control-airbyte-configurations).
r
We use octavia-cli and an external password vault that replaces the password variables. When octavia-cli publishes said connectors it automatically replaces them with the env vars as shown in the documentation. Makes for a very easy GitOps based deployment strategy in airbyte. So firstly we create the connector within airbyte, then back it up into the octavia init'd repo. Replace the password/key/etc. values in the configuration.yml with the env vars and our actions do the rest for deployment.