Hi! We are in the process of setting up Airbyte Docker via Pulumi on a GCP compute engine and there are some open questions: • How would we persist state in case of updating the docker image or replacing the instance? I saw this ticket (https://github.com/airbytehq/airbyte/issues/3605) for setting up an external config db, but one reply suggests, this won't be enough for persisting state? • What would you recommend for setting up the initial config with basic sources and destinations. Are there options to do this via Pulumi as well? Thanks for your help!

1. You can use an external database for this OR export/import the config. 2. You can create sources/destination using the UI or the API, but not with Pulumi. There is some work to start using Terraform, maybe in the future we can extend to Pulumi

Thanks @[DEPRECATED] Marcos Marx!

Hi @Andreas, I am in the process of setting up the exact same thing. Ideally a Airbyte via Pulumi on a GCP compute engine with an external GloudSQL Postgres. Did you manage to do it ?!

Hi! I am at a different company now and we are not using pulumi but terraform. We set up a a compute engine where the docker image is running and we are using cloud sql database to persist state. Four managing the connectors, we are using a third-party terraform module.

Hi @Andreas, I am really unclear on how to proceed. I am using pulumi to create the VM instance and the cloudSQL postgres instance. On this link: https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine We see that we can launch airbyte using

run-ab-platform.sh

On this one: https://docs.airbyte.com/operator-guides/configuring-airbyte-db , to connect to an external database, they explain we should modify the .env file After some digging, I see some users talking about using this "maneuver" but encountering errors, this way I might change the env variables inside the .env file before launching the docker compose.

mkdir airbyte && cd airbyte
wget <https://raw.githubusercontent.com/airbytehq/airbyte-platform/main/{.env,flags.yml,docker-compose.yaml}>
docker compose up

How did you guys manage to do it, even if it's terraform the concept should be a reproducible.

You can set a startup script on the compute engine via pulumi. With that you can download the mentioned files to the machine. Then you will have to either overwrite some of the environment variables in the .env file with the values that result from your pulumi deployment (config for the cloudsql db) or set environment variables in the bash script directly. After that you can do the docker compose up. All in the same startup script.

thank you for the response @Andreas 💪