Hey folks, In order to catalog the Airflow DAGs in our DataHub, should I follow https://datahubproject.io/docs/metadata-ingestion/#lineage-with-airflow? I just wonder if there is any simple recipe that I can ingest DAGs from Airflow without interfering with Airflow hooks and operators? As far as I know, Airflow already exposes a REST API for this metadata. (e.g. /api/v1/dags, /api/v1/dags/{dag_id}). Maybe, we can utilize that endpoint. What do you think?

Hey there @polite-flower-25924 - yes, a source like what you are describing could be written. The advantage of the lineage backend approach is that airflow manages all the logistics of finding the dags and tasks, so Datahub doesn't need to worry about things like pagination. The rest API also makes it harder to understand how often dags are running.

Thank you @green-football-43791! It makes sense. Then, I’m going to integrate Airflow with Lineage backend 🙂 Let’s see how it is going.