A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi All! On a non Neo4j deployment, I saw datahub stored lineage in MySQL tables as upstreamLineage aspect.. Are there there other places where the lineage is also stored? I deleted all upstreamlineage rows from the backend database, but i still see Lineage in DataHub ui...

hey <@U02GFNXF5R9> - in non-neo4j deployments lineage data is indexed in elasticsearch

this allows datahub to look up lineage in both direction performantly.

if you run the restore indices command, you'll see elastic catch up with mysql

In general, deleting rows directly from MySQL will lead to unpredictable results. It is better to use <https://datahubproject.io/docs/how/delete-metadata/#rollback-ingestion-batch-run|rollbacks> if you want to undo certain ingestion runs.

We will keep evolving the storage layer and you may break your installation if you delete the wrong rows

Thanks! <@U01JUBAPZPS> &amp; <@U01C3DMG2GL>! I had my upstream lineage created by posting multiple MCEs which were not a part of ingestion job.. I guess i need to wrap my MCEs in ingestion pipeline..