I have a question regarding the integration of Prefect and Airbyte.
I have a Prefect flow that is copying a couple of Google BigQuery tables between 2 different GCP projects.
The flow is reading a config (json) file that I have in google storage.
For each region and table in the config file a copy task is spawned using Prefect map option (I have multiple regions, each one has different tables - each table has its own unique source and destination).
Now I was wondering if its possible to do the same thing using Airbyte? spawning a new connection by using Prefect if this connection doesn’t exists (lets say - for region A between BQ table a and table b) and if it does exist - just trigger the sync?
More specifically - since my configuration is dynamic and regions/tables will get added I don’t want to configure each connection manually in the Airbyte UI.
Augustin Lafanechere (Airbyte)
01/25/2022, 2:15 PM
Hi @Dekel R, I don't think that Prefect is capable of creating Airbyte connections (I'm not familiar with Prefect so feel free to correct me if I'm wrong). We are working on a CLI project that could help you programmatically create connections according to your configuration with some scripting. If you're interested I suggest you to join #public-octavia-cli to receive updates on this project.
01/30/2022, 6:47 AM
Seems like the CLI project is exactly what I need.
I want to use the syncing features of Airbyte and the ability to connect different data sources - I just can’t use the UI since they are constantly changing in my case.
Is there any timeline or a road map on this?
01/31/2022, 5:28 AM
Hey I think you can post the questions around the cli over the channel and yeah there is surely work which is in progress.