A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi guys. I do ingest lineage using cli, that is working good. but i can’t deleting it. how can i delete lineage data?

Right now, the best bet is to emit an empty lineage aspect for that entity.

cc <@U02DMDKSZME> - is there a way to express “no lineage” in your yaml format? If not, I could see it being pretty helpful to add that option so folks can easily clear their data out.

Unfortunately no, if there's no upstream specified for a given entity then it will <https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/metadata/lineage.py#L120|skip emitting a MCP> for it 

But if you're looking to just hard replace some up streams you have then yeah this source could work. Just need to set `preserve_upstream: False` in your recipe

<@U02DMDKSZME> do you think it would make sense to provide an option, like `[]` to clear out lineage?

Maybe we could put that up as a request for contribution task

Yeah I can see an argument for it, when I was writing it I wanted to avoid accidentally removing lineage aspects from entities unless they had another one in mind

But yeah currently it's not super easy to remove that aspect anyway

Yea makes sense-- I think this will become more of a need now that you’ve made it so much easier to add lineage :smile: this is the 2nd question about removing lineage I saw come up yesterday

Ha- we are definitely moving in the right direction at least!

better than folks not having any lineage at all :joy:

Hey <@U02DMDKSZME>! Just stumbled across this thread — do you have capacity to contribute back something like Gabe’s suggestion? Or some other way for folks to remove lineage edges that are specified via yaml? it sounds super useful! <https://datahubspace.slack.com/archives/C029A3M079U/p1646925899981719?thread_ts=1646898865.642549&amp;cid=C029A3M079U>

For sure, i'll probably have some bandwidth with it in April, im actually spending this month productionalizing our datahub setup this month :stuck_out_tongue: so a bit distracted atm. It should be a good first-issue if a new contributor wants to tackle it before i get to it