Hey there! I wanted to version control my sources, destinations, and connections. But then I saw that airbyte switched away from files and stores the configuration in PG https://github.com/airbytehq/airbyte/blob/master/docker-compose.yaml#L22 which makes it harder to version control. I'm wondering if airbyte a) offers a utility like

dump-airbyte-configuration-to-file

and b) the option to load that file when running airbyte via

docker-compose up

Or maybe you folks have another nice workaround.

@Stefan Otte, you can still use the API to export the configs to file and version control the output.

API?! Do you mean this one? https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html After exporting it can I also just reload it?

take a look at the bottom of this page: https://docs.airbyte.io/operator-guides/upgrading-airbyte

Thanks

If I understand it correctly:

# Export config
curl -H "Content-Type: application/json" \
    -X POST localhost:8000/api/v1/deployment/export \
    --output /tmp/airbyte_archive.tar.gz

# Import config
curl -H "Content-Type: application/x-gzip" \
    -X POST localhost:8000/api/v1/deployment/import \
    --data-binary /tmp/airbyte_archive.tar.gz

I would suggest to make this even easier or at least document it more prominently. Version controlling your config should be one of the first things people do when setting up ELT.

And plaintext configs would be even nicer to version control. (I know you went for PG on purpose, I saw the ticket 🙂)

I should also say: except the config that is a bit hard to version control and not being able to deselect columns to sync (I also saw the tickets for that feature) I can only say positive things about airbyte. Really great tool and nice onboarding 👍

Thanks @Stefan Otte, that’s good ideas to write about indeed cc @abhi / @Ari Bajo

along the lines of version control, what does a development life cycle look like? e.g. if there's a new table to be ingested from an existing mysql source, is the recommendation that no pre-prod testing is needed (e.g. with binlog approach)? cc: @Atif Imam

@Stefan Otte Hi, i’m curious what you have eventually done to version control your configurations. I’m struggling for the same problem

@Jingkun I ended up not using aribyte because I can't easily exclude columns when extracting/loading. But the snippet that I posted earlier should allow you to at least import/export the config and you can version control that. Not perfect, but at least you can version control something.

gotcha, thank you

Is there a way to only export a specific airbyte connection (to version control it)? My local aribyte instance (docker) shows all the the connections that I created in the past. I just want to export and version control a specific one. Or alternatively: how can I have project specific airbyte instances with docker? CC @Chris (deprecated profile) @Liren Tu

when i try the export way, my credentials are all clear text in the config files. Is there a way to export with hashed credentials so I can commit it to git?

Is there a way to only export a specific airbyte connection (to version control it)?

Yes. You can use the connection API: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/get

how can I have project specific airbyte instances with docker?

You can do this with multiple workspaces in the same instance. Currently the OSS version does not support multiple workspaces in the UI. But you can still create and manage them through the API: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/workspaces/create