What is the best practice for upgrading Airbyte? The official instructions are as follows: Remove previous local

docker-compose.yaml

and

.env

./run-ab-platform.sh -r

Bring Airbyte back online, optionally with the

-b

flag to run the containers in the background (docker detached mode).

./run-ab-platform.sh -b

This deletes your .env file, which contains configuration of for example the Airbyte database. Since one of the steps just deletes you .env file, are you supposed to manually edit the newly downloaded one? Without it you can’t deploy…. It doesn’t seem to make a lot of sense.

The process of upgrading Airbyte involves several steps, and it's important to follow them in the correct order to ensure a successful upgrade. 1. First, you need to turn off Airbyte. This can be done by running the command

docker compose down

in the terminal on the host where Airbyte is running. 2. Next, you upgrade the Docker instance to the new version. If you're running Airbyte from a cloned version of the Airbyte GitHub repo and want to use the most recent stable version, you can simply run

git pull

. If you're running Airbyte from downloaded

docker-compose.yaml

and

.env

files without a GitHub repo, you can run

wget <https://raw.githubusercontent.com/airbytehq/airbyte/master/run-ab-platform.sh>

to download the installation script. 3. After that, you remove the previous local

docker-compose.yaml

and

.env

files by running

./run-ab-platform.sh -r

. 4. Finally, you bring Airbyte back online by running

./run-ab-platform.sh -b

. You can add the

-b

flag to run the containers in the background (docker detached mode). Regarding your concern about the

.env

file, it's important to note that this file is regenerated when you run the

./run-ab-platform.sh -r

command. This means that any custom configurations you had in your previous

.env

file will need to be manually added to the new

.env

file after the upgrade. If you're using Airbyte on Kubernetes, the upgrade process is slightly different and involves exporting your data, turning off Airbyte, deleting the existing Airbyte Kubernetes volumes, checking out the most recent version of Airbyte, bringing Airbyte back up, and importing your migrated archive. For more detailed instructions, you can refer to the [official Airbyte documentation](https://docs.airbyte.com/operator-guides/upgrading-airbyte).

So the best practice is exactly like I feared, very manual. This should be improved.

Depending on how you are deploying you could use cloud-init/userdata to handle templating out the configs. That is what we do here. ALB -> EC2 instance with userdata templated via terraform to deploy our airbyte services.