@kapa.ai I want to migrate the internal PostgreSQL database used by Airbyte (deployed via Helm) to an external PostgreSQL instance on AWS RDS. What steps do I need to follow to safely move all the data and avoid any inconsistencies in Airbyte?

To migrate Airbyte’s internal PostgreSQL database (deployed via Helm) to an external PostgreSQL instance on AWS RDS, follow these steps to ensure a safe and consistent migration: --- ### 1. Prepare the External PostgreSQL Instance - Create the required databases on your AWS RDS instance. Airbyte requires at least three databases:

airbyte

,

temporal

, and

temporal_visibility

. You can create them with:

sql     CREATE DATABASE airbyte WITH OWNER = <your_user>;     CREATE DATABASE temporal WITH OWNER = <your_user>;     CREATE DATABASE temporal_visibility WITH OWNER = <your_user>;

Replace

<your_user>

with the user you plan to use for Airbyte ([source](https://github.com/airbytehq/airbyte/issues/49304)). --- ### 2. Backup the Internal Database - Stop Airbyte services to prevent new data from being written during the migration. - Dump the contents of each database from your current (internal) PostgreSQL instance:

bash     pg_dump -h <internal_host> -U <user> airbyte > airbyte.sql     pg_dump -h <internal_host> -U <user> temporal > temporal.sql     pg_dump -h <internal_host> -U <user> temporal_visibility > temporal_visibility.sql

([source](https://github.com/airbytehq/airbyte/issues/49304), [migration example](https://discuss.airbyte.io/t/3514)) --- ### 3. Restore the Data to AWS RDS - Restore each dump to the corresponding database on your RDS instance:

bash     psql -h <rds_host> -U <user> -d airbyte -f airbyte.sql     psql -h <rds_host> -U <user> -d temporal -f temporal.sql     psql -h <rds_host> -U <user> -d temporal_visibility -f temporal_visibility.sql

([source](https://github.com/airbytehq/airbyte/issues/49304)) --- ### 4. Update Airbyte Helm Configuration - Create a Kubernetes Secret with your database credentials (if not already done):

yaml     apiVersion: v1     kind: Secret     metadata:       name: airbyte-config-secrets     type: Opaque     stringData:       database-user: <your_user>       database-password: <your_password>

([source](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/database)) - Edit your

values.yaml

to disable the internal database and point to your external RDS instance:

yaml     postgresql:       enabled: false      global:       database:         type: external         secretName: "airbyte-config-secrets"         host: "<rds_host>"         port: "5432"         database: "airbyte"         userSecretKey: "database-user"         passwordSecretKey: "database-password"

([source](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/database)) --- ### 5. Deploy Airbyte with the New Configuration - Apply the updated Helm chart. If you encounter timeouts, try increasing the Helm install timeout (e.g.,

--timeout 10m0s

) as initial migrations can take time ([source](https://github.com/airbytehq/airbyte/issues/48542)). --- ### 6. Validate the Migration - Start Airbyte and verify that all connections, jobs, and configurations are present and working as expected. - Check logs for any errors related to database connectivity or schema mismatches. --- ### Important Notes - Manual migration is required: Airbyte does not automatically migrate internal database data to an external database. You must perform the backup and restore steps manually ([source](https://discuss.airbyte.io/t/2903)). - Version compatibility: Ensure your RDS PostgreSQL version is compatible with Airbyte. Community reports show success with versions up to Postgres 17.1 ([source](https://github.com/airbytehq/airbyte/issues/48542)). - Downtime: Plan for downtime during the migration to avoid data inconsistencies. --- If you follow these steps, you should be able to safely migrate your Airbyte internal database to an external AWS RDS instance.