Hey everyone! I have a question regarding the integration of Prefect and Airbyte. I have a Prefect flow that is copying a couple of Google BigQuery tables between 2 different GCP projects. The flow is reading a config (json) file that I have in google storage. For each region and table in the config file a copy task is spawned using Prefect map option (I have multiple regions, each one has different tables - each table has its own unique source and destination). Now I was wondering if its possible to do the same thing using Airbyte? spawning a new connection by using Prefect if this connection doesn’t exists (lets say - for region A between BQ table a and table b) and if it does exist - just trigger the sync? More specifically - since my configuration is dynamic and regions/tables will get added I don’t want to configure each connection manually in the Airbyte UI. Thanks!!

Hi @Dekel R, I don't think that Prefect is capable of creating Airbyte connections (I'm not familiar with Prefect so feel free to correct me if I'm wrong). We are working on a CLI project that could help you programmatically create connections according to your configuration with some scripting. If you're interested I suggest you to join #C02RRUG9CP5 to receive updates on this project.

Hey, Seems like the CLI project is exactly what I need. I want to use the syncing features of Airbyte and the ability to connect different data sources - I just can’t use the UI since they are constantly changing in my case. Is there any timeline or a road map on this? Thanks!!

Hey I think you can post the questions around the cli over the channel and yeah there is surely work which is in progress.