Hello team, I just hear about Airbyte some days ago, but cannot find anything describe the relationship between Airflow & Airbyte. I'm imaging that we should create an Airflow DAG to enable an Airbyte Job, and this job will run by Airbyte Scheduler & Worker. However if we do something like that, we will have Airflow Webserver, Scheduler, Worker & Airbyte Webserver, Scheduler, Worker. Do we have redundancy Scheduler & Worker? Or doing something like that will have some pros that I didn't notice?
thank you, I will try it and feedback with you if it will help or not ❤️
Please correct me if I'm wrong, above document is exactly what I'm imaging about the relationship between Airflow & Airbyte, but what i'm mention here is about the redundancy Scheduler & Worker if we do something like that. Do you have any idea about that, maybe it have some pros (& cons)?
[DEPRECATED] Marcos Marx
11/04/2021, 8:41 PM
Thien, Airflow is a different project so we cannot use their scheduler/worker. If you want to have high-availability of Airbyte you can deploy using Kubernetes.