https://linen.dev logo
#feedback-and-requests
Title
# feedback-and-requests
s

Stefan Otte

10/01/2021, 4:18 PM
Hey there! I wanted to version control my sources, destinations, and connections. But then I saw that airbyte switched away from files and stores the configuration in PG https://github.com/airbytehq/airbyte/blob/master/docker-compose.yaml#L22 which makes it harder to version control. I'm wondering if airbyte a) offers a utility like
dump-airbyte-configuration-to-file
and b) the option to load that file when running airbyte via
docker-compose up
Or maybe you folks have another nice workaround.
@Liren Tu may have a plan for this!
u

user

10/01/2021, 4:20 PM
@Stefan Otte, you can still use the API to export the configs to file and version control the output.
u

user

10/01/2021, 4:27 PM
API?! Do you mean this one? https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html After exporting it can I also just reload it?
u

user

10/01/2021, 4:28 PM
take a look at the bottom of this page: https://docs.airbyte.io/operator-guides/upgrading-airbyte
u

user

10/01/2021, 4:34 PM
Thanks
u

user

10/01/2021, 4:35 PM
If I understand it correctly:
Copy code
# Export config
curl -H "Content-Type: application/json" \
    -X POST localhost:8000/api/v1/deployment/export \
    --output /tmp/airbyte_archive.tar.gz

# Import config
curl -H "Content-Type: application/x-gzip" \
    -X POST localhost:8000/api/v1/deployment/import \
    --data-binary /tmp/airbyte_archive.tar.gz
u

user

10/01/2021, 4:36 PM
I would suggest to make this even easier or at least document it more prominently. Version controlling your config should be one of the first things people do when setting up ELT.
s

Stefan Otte

10/01/2021, 4:37 PM
And plaintext configs would be even nicer to version control. (I know you went for PG on purpose, I saw the ticket 🙂)
u

user

10/01/2021, 4:43 PM
I should also say: except the config that is a bit hard to version control and not being able to deselect columns to sync (I also saw the tickets for that feature) I can only say positive things about airbyte. Really great tool and nice onboarding 👍
u

user

10/01/2021, 5:29 PM
Thanks @Stefan Otte, that’s good ideas to write about indeed cc @abhi / @Ari Bajo
u

user

10/08/2021, 8:20 AM
along the lines of version control, what does a development life cycle look like? e.g. if there's a new table to be ingested from an existing mysql source, is the recommendation that no pre-prod testing is needed (e.g. with binlog approach)? cc: @Atif Imam
u

user

10/12/2021, 3:38 PM
@Stefan Otte Hi, i’m curious what you have eventually done to version control your configurations. I’m struggling for the same problem
u

user

10/13/2021, 6:46 AM
@Jingkun I ended up not using aribyte because I can't easily exclude columns when extracting/loading. But the snippet that I posted earlier should allow you to at least import/export the config and you can version control that. Not perfect, but at least you can version control something.
u

user

10/13/2021, 1:06 PM
gotcha, thank you
u

user

10/14/2021, 8:17 AM
Is there a way to only export a specific airbyte connection (to version control it)? My local aribyte instance (docker) shows all the the connections that I created in the past. I just want to export and version control a specific one. Or alternatively: how can I have project specific airbyte instances with docker? CC @Chris (deprecated profile) @Liren Tu
u

user

10/14/2021, 9:17 PM
when i try the export way, my credentials are all clear text in the config files. Is there a way to export with hashed credentials so I can commit it to git?
u

user

10/16/2021, 9:27 AM
Is there a way to only export a specific airbyte connection (to version control it)?
Yes. You can use the connection API: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/get
u

user

10/16/2021, 9:28 AM
how can I have project specific airbyte instances with docker?
You can do this with multiple workspaces in the same instance. Currently the OSS version does not support multiple workspaces in the UI. But you can still create and manage them through the API: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/workspaces/create
2 Views