A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi team, :slightly_smiling_face: I was trying to ingest `dbt` on bigquery... until i realized ingesting bigquery now supports lineage information. So, I was testing out to use lineage information only out of bigquery, after hard deleting all dbt entities. However, When i do it.. dbt entities(=nodes) still appear in the lineage graphs like haunting ghosts .. How can i REALLY delete them ? :slightly_smiling_face:

I tried deleting the data from database itself, but still i can see the entities.

Uh oh! Have you tried running a datahub delete using the cli?

As discussed here: <https://datahubproject.io/docs/how/delete-metadata/#delete-by-urn>

Yes, i hard deleted using`datahub delete -p dbt --hard`

maybe datahub doesn't recognize dbt as platform ?

Funny thing is, i can't see dbt on the main page, but it appears on the lineage

I see -can you try also adding the "dataset" entity filter?

Also- did you try running this before or after you manually deleted from your DB?

First i tried CLI , then i tried deleting from DB

I just tried datahub delete -p dbt --entity_type dataset --hard, but it still captures 0 entities

I just figured out, when i run CLI commands , data only disappear on UI but still remains on DB (i also tried on bigquery entities.)

So that means its removed from indexes but not from the MySQL db. This is definitely not expected. Is there an ingestion run you can try to rollback, instead of using these filters?

I have seen similar inconsistencies as well - when doing a hard delete from the CLI the data from MySQL was deleted fine, but the data on Elasticsearch for _some_ datasets was still there (let's say, 7 out of 50 - 43 datasets were properly deleted from both storage databases and 7 only deleted in MySQL). So the datasets would be counted in the UI but when I clicked on them there would be no information. When I deleted them manually from the DB they'd then disappear also from the UI

<@UV0M2EB8Q> <@U01JUBAPZPS> You may want to take a look here. This is pretty odd. Is there any pattern you noticed in the ones remaining? Also, did this occur when you deleted by run_id (normal rollback) or using the filter API? (by platform, entity type, etc)

Hi <@U02EXJZ0KMM> and <@U02LNBFDUP7>, I have tried to repro  the issue on the latest master using default snowflake and bigquery recipes, with different patterns of ingests + deletes. Both mysql and elastic index row counts are exactly the same after an ingest + delete, as they were before. A local repro is still elusive and looks like something else must happen to have stale entities left after a hard deletion of a platform. Would you mind sharing what kind of residual data(the entity/aspect types) you still see lingering in elastic?

Hi <@U028L1V9BE1> - sorry for the delayed response on this, was on leave + some other work commitments had taken over. But, i have not forgotten about this, when we see the issue happening again I'll make sure I share screenshots, logs etc