We shared a new Data Insights Post about Data Orchestration. It mainly covers these topics:
• The Shift from Data Pipelines to Data Products
• Data-aware Pipelines that know about the inner life of a task
• Why a declarative approach with higher-level abstractions helps
• How abstractions improve reuse code between complex cloud environments
• Where do we come from with the evolution of Data Pipeline Orchestration
• Open-source data orchestration tools such as Airflow, Prefect, Dagster
Let us know what you think. Comments and discussions are welcome :).
https://airbyte.com/blog/data-orchestration-trends