Hi everyone, we got an Airbyte instance spun up on a VM to serve as a test run. After some tinkering, everything is working well (even made a few of our own custom connectors!). We’re ready to migrate to a K8 instance with a standalone Postgres database. Everything is connected and a new instance is set up but we want to transfer the connections and states from the VM to the new K8 instance. Is this possible and what tables should be moved over? Any recommendations methods for doing this?

Hi Kevin if you want to to migrate your connectors and config you can copy

airbyte_db

volume from your test instance and use it on your new one... you can find it on

var/lib/docker/volumes

🙂

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2572 to ensure timely resolution!

But I’m moving from the DB being in Docker to a stand-alone Postgres db. Transferring the volume want work in that instance.

• hey Kevin I have a question for @Marcos Marx (Airbyte), I have 2 instances with airbyte and one is running on version

0.40.4

and the other is running version

0.40.10

on the first instance (

0.40.4

) I have the option to "export" my configuration (see pic 1). Do you have that? It's under settings/configuration. if you do, you can do and export and then upload to your new instance and have all you need. The question I have for @Marcos Marx (Airbyte) is that on the newer version

0.40.10

I don't see that option anymore, did Airbyte took this option out? (see pic 2).

I had a similar case and I migrated using the export configuration mentioned by Caio. You can also just dump the current database and recover it on the standalone postgres

@Caio César P. Ricciuti, I was hoping for the same export functionality but it was migrated out in

.040.7

I believe to close a security gap. There doesn't seem to be an easy way to reenable it. I've just done a

pg_dump

from the Docker container and am working to get that imported into the standalone. It has been rocky thus far given difference configurations and user privileges. I'll report back on progress.

@Kevin Peters, you may try to downgrade just the airbyte-webapp, so you will have access to that UI

@Caio César P. Ricciuti https://github.com/airbytehq/airbyte/pull/16175 it was removed because could cause security issues. Some users where deploying Airbyte on public and removing this feature was to help them

OK, just to report back, I was able to migrate the database from Docker to a standalone instance with a little trial and error. Here are the final steps: 1. Pause all connections on the Airbyte instance. 2. Export the database from docker as a series of INSERT statements. Use

docker exec airbyte-db pg_dump -U docker  --column-inserts --data-only  airbyte > airbyte.sql

to create the file. 3. Get the new Airbyte instance set up and connected to the standalone database. (You could also just connect the old instance to the new database if you'd like) 4. Run through the normal set up process so that the tables are created but don't create any sources or destinations. 5. I got the .sql file off the VM by setting up a quick SFTP drive. 6. Connect to the standalone postgres server so that you can run SQL commands directly. I used a visual editor. 7. Update the Workplace table to match the values in your original database; I pulled the values from my SQL file. I then deleted this insert statement from the sql file. 8. I then deleted all records from the

actor_catalog

table (They'll be re-added in a second). This ensured my custom connectors transferred too. 9. Finally, I executed the full SQL statement which migrated the docker database to the standalone. Total time was 20 minutes or so. It honestly took me WAY longer to figure out how to connect my Google Kubernetes instance to the Cloud SQL server. (That was ridiculous.)

awesome, thanks for the debrief 👍 Good to know that the identifiers for Airbyte to initialize properly are all contained within the Workplace table, and that it has to be set up first.

@Marcos Marx (Airbyte) there is going to be another way to keep a backup (as before), from airbyte connections/sources/destinations? Or now I need to backup by my self (something like a airbyte_db volume backup)?

You need backup yourself 😞