Hi, I got one very simple question in AirByte architecutre.
If Airbyte already has a scheduler and workers, why does it need Airflow? What's the point of triggering Airbyte job from Airflow?
01/24/2022, 12:46 PM
Airflow makes sense when your Airbyte data source is part of a bigger picture with multiple data sources, transformations in dbt, data quality validation etc.
Augustin Lafanechere (Airbyte)
01/24/2022, 1:21 PM
Hi @Lihan, I confirm Jonas explanation. If you are using Airbyte for the full ELT stack, triggering DBT transformation from Airbyte, you don't need an extra orchestrator. But in case you need to trigger transformation jobs after an Airbyte replication that are not run with DBT, an orchestrator such as Airflow is what you need.