Hey Team, I am looking at deploying Airbyte using ...
# replication-ask-ai
c
Hey Team, I am looking at deploying Airbyte using Rancher and connection to an external Postgres Database. Does anyone have experience with this? As I will be deploying it to run using Rancher v1.5.1 i think the best way is to dissect the docker-compose and deploy each image as separate services. I will also be looking at scaling up the workers as needed hence the reason for this setup. Does anyone have advice?
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet about deploying Airbyte using Rancher specifically. However, I can provide some information on connecting Airbyte to an external Postgres database. To configure Airbyte services with an external Postgres database, you need to edit the following environment variables declared in the
.env
file:
Copy code
DATABASE_USER=postgres   DATABASE_PASSWORD=password   DATABASE_HOST=host.docker.internal # refers to localhost of host   DATABASE_PORT=3000   DATABASE_DB=postgres
You can also separate the Config Database and the Job Database by specifying separate parameters:
Copy code
CONFIG_DATABASE_USER=airbyte_config_db_user   CONFIG_DATABASE_PASSWORD=password
Additionally, you must redefine the JDBC URL in the environment variable
DATABASE_URL
to include the correct host, port, and database. If you need to provide extra arguments to the JDBC driver (for example, to handle SSL), you should add it here as well:
Copy code
DATABASE_URL=jdbc:<postgresql://host.docker.internal:3000/postgres?ssl=true&sslmode=require>
For the config database, if it is separate from the job database:
Copy code
CONFIG_DATABASE_URL=jdbc:postgresql://<host>:<port>/<database>?<extra-parameters>
(Source: [Configuring the Airbyte Database](https://docs.airbyte.com/operator-guides/configuring-airbyte-db)) Please note that when connecting to an external Postgres database, you need to manually backup the database, as Airbyte doesn't have a migration feature between the internal database ([source](https://discuss.airbyte.io/t/2903)). For more information on deploying Airbyte on Kubernetes, you can refer to the [official documentation](https://docs.airbyte.com/deploying-airbyte/on-kubernetes).
c
thanks yeah I have created two databases for jobs and config in external postgres.