A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi Team,
Facing an issue with publishing Lineage Information.Whenever I publish Lineage ( between data job and dataset) , I can see the information in the UI. but after the DAG runs, the lineage information is lost.

Hi <@U029VFMT1GV>, what ingestion source are you having trouble with here? Could you post the logs?

Note that if the source you are running is emitting Lineage, then there is a possibility that the Manual Lineage will conflict with the ingestion source. In those cases, we accept the ingestion source's picture of Lineage because it is the assumption that technically inferred lineage is more often correct

<@U01GCJKA8P9> - The Source( here Airflow pipeline ) is running on a schedule ( daily ). The lineage we added are not overriding the source ( we are adding additional information on which task is creating which dataset ).
But this is lost each time the source emits lineage

<@U01GCJKA8P9> - The Source( here Airflow pipeline ) is running on a schedule ( daily ). The lineage we added are not overriding the source ( we are adding additional information on which datajob or task is creating which dataset entity).
But this is lost each time the source emits lineage. Is this the expected behavior ?
 Let me know if you need any more details ?

<@U01GCJKA8P9> - Is there a suggestion for this scenario, when we have backend emitting lineage and also include lineage for the datasets and tasks , how to have both of them included ?

Hi <@U029VFMT1GV> I understand the problem clearly, now. Basically Airflow ingestion emits the Task -&gt; Task relationships, and you want to augment the Task -&gt; Dataset relationships in the UI.

Currently, this is not possible, as the Airflow run will overwrite _both_ the Task -&gt; Task edges and the Task -&gt; Dataset edges. There is a possible solution (patch emission from the Airflow producer), but that has not yet been implemented.

This is not a good user experience, and we are working on our side to consider what to do in such cases. For now, I would recommend trying to manage the Task -&gt; Dataset relationships on Airflow using Inlets and Outlets supported by Airflow's lineage backend. I know this isn't ideal but it's the only way to avoid conflict today.

cc. <@U03BEML16LB> <@U01GZEETMEZ> for awareness!

<@U01GCJKA8P9> - Is there a feature request for Patch emission for Airflow ? If not would like to create one.

Hi <@U029VFMT1GV> I don’t believe there is- you can create one here <https://feature-requests.datahubproject.io/>

<@U01JUBAPZPS> - This is the issue we are referring to. Any ideas on the Patch Emission for Airflow plugin ?