i have a question regarding airflow lineage - https://datahubproject.io/docs/metadata-ingestion/#lineage-with-airflow. Is this the way to do this?

Yes. That’s how I do and it’s the recommended way as far as I know. Prerequisite: The airflow version must be 1.10.15+ or 2.0.2+ • Download acryl-datahub[airflow] package. • Add datahub GMS Rest endpoint to airflow connections • Enable DatahubLineageBackend in

airflow.cfg

• Trigger lineage_backend_taskflow_demo.py

thanks, what are we gaining by adding the DatahubLineageBackend in airflow.cfg, isnt the DAG manually sending the lineage to gms?

DAG does its own job. It doesn’t manually send data to DataHub. Airflow triggers the lineage backend to send the lineage data to DataHub with

inlets

and

outlets

parameters that creates the lineage.

Do datasets declared in inlets/outlets have to be ingested beforehand into Datahub? Can you define a task as an inlet?

i dont think so, if the dataset does not exist, it will create an empty dataset IIRC

The dataset does not have to exist, but you cannot put a task as an inlet as of now.

@brief-cricket-98290, if you are referring to airflow tasks as an inlet, no you cannot. Airflow tasks accept inlets and outlets to capture dataset lineage. The datahub example above leverages this airflow feature to capture the dataset lineage. See the airflow documentation for more details.

Noted, thank you all!

One follow up question - We are dealing with Redshift and Dbt operators. Is there a way where we can identify the inlets and outlets automatically instead of hardcoding values?