Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hello team, I just hear about Airbyte some days ago, but cannot find anything describe the relationship between Airflow &amp; Airbyte. I'm imaging that we should create an Airflow DAG to enable an Airbyte Job, and this job will run by Airbyte Scheduler &amp; Worker. However if we do something like that, we will have Airflow Webserver, Scheduler, Worker &amp; Airbyte Webserver, Scheduler, Worker. Do we have redundancy Scheduler &amp; Worker? Or doing something like that will have some pros that I didn't notice?

<https://docs.airbyte.io/operator-guides/using-the-airflow-airbyte-operator>

thank you, I will try it and feedback with you if it will help or not :heart:

Please correct me if I'm wrong, above document is exactly what I'm imaging about the relationship between Airflow &amp; Airbyte, but what i'm mention here is about the redundancy Scheduler &amp; Worker if we do something like that. Do you have any idea about that, maybe it have some pros (&amp; cons)?

Thien, Airflow is a different project so we cannot use their scheduler/worker. If you want to have high-availability of Airbyte you can deploy using Kubernetes.