Hi guys, I'm using Datahub as lineage backend of GCP Composer(Airflow). I found something strange. 1. Run a DAG which have 3 tasks. so Datahub shows 3 tasks for the DAG(Pipeline) and its lineage 2. Remove 2 tasks in the DAG and re-run 3. Expect: Datahub shows 1 task for the DAG and updated lineage 4. Actual: Datahub still show 3 tasks and their lineage Which component or configuration should I check? Is it normal?

Hi @adorable-flower-19656, that's the expected behavior in the current implementation. The lineage is captured via the DataJobInputOutput aspect as you can see from here. The new run completely overwrites the older value of the aspect, leading to the observed behavior.

Hi @helpful-optician-78938, I didn't understand it. As you said, if the new run completely overwrites the older value, my Datahub should have shown 1 task information. But there was no any change. Am I misunderstood?

Sorry for the confusion, I mixed up actual & expected. I think there is some stale metadata at play here. Please log an issue and I'll look into it asap.

Thanks for your reply @helpful-optician-78938 🙂 I'm going to clean all my metadata and re-produce the issue. And then I'll log here.

@helpful-optician-78938 is this issue fixed? I still see this in 0.8.44