https://datahubproject.io logo
Join Slack
Powered by
Hi, team! What are the best practices to delete da...
# ingestion
h
Hi, team! What are the best practices to delete dataset to dataset lineage? Also, is there any way to delete datasets other than Datahub CLI?
a
I have deleted lineage by emitting an empty array
s
You can use the CLI to delete the aspect 
upstreamLineage
for the dataset.
h
Ok, great. And no other options to delete other than CLI?
s
Are you facing some issue with the CLI?
You can go into mysql, delete the aspect rows, re-index elastic. That will also get things. But CLI would be simpler
h
I was looking for options like curl, for example.
b
To avoid having to install the python library sounds pretty useful
s
you can avoid installing the python library and just use the docker image to run the CLI https://datahubproject.io/docs/cli/#using-docker
h
If we want some other application to send this kind of data to datahub, forcing them to install a library or running another docker image is not user friendly at all.
s
The endpoints are there. Let me check what curl request would work
b
it should be possible since the rest emitter has a make_curl_command inside
h
Hmm, indeed, I didn't notice that
s
Give me a few minutes. I will get you the curl for delete
🙌 1
Copy code
curl "<http://localhost:8080/entities?action=delete>" -X POST --data '{"urn": "urn:li:dataset:(urn:li:dataPlatform:hive,fct_users_deleted,PROD)"}'
Something like this should work
Note this is hard-delete from mysql
Adding this in docs for future ref https://github.com/linkedin/datahub/pull/3928
h
Great, it works. Any chance you also have a command to delete a particular aspect? I could not find that in the documentation
s
Let me ask someone. I am not really sure
We don't have a command to delete a specific aspect.
h
I had 7 upstreams for a dataset, next time this dataset had only 5 upstreams, and after I ingested them I still had initial 7 upstreams. Even if I ingest an empty array, they don't get deleted. Looks like it works in append-only mode, not replacing anything. So I guess the only working option is to delete and reingest the dataset, losing all stats and edited information.
s
Another option is to do a re-indexing
with clean option mentioned in docs
h
I'll try that
I've tried instruction here - https://datahubproject.io/docs/how/restore-indices/ And it did not help. Don't know what's wrong
s
with the 
-a clean
option?
h
Yes
l
Hi @handsome-belgium-11927! I believe this has been fixed; can you confirm if you’re still having issues on your end? (cc: @green-football-43791)
h
Hi, Maggie. What has been fixed? Reindexing? Or the ingestment of another ammount of upstreams?
g
@handsome-belgium-11927 what version of datahub are you running?
We fixed an issue in the index logic to deal w/ deleting all the edges of an entity
h
Ok, I'll try it tomorrow. Though this is not a convenient way to delete lineages, and I hope reingesting a dataset with actual upstreams would replace previous upstreams, rather than merging this two lists.
g
yep- it should
re-ingesting a dataset w/ different upstreams should replace previous upstreams
i would check the datahub version you are on if you are seeing that
it would be a surprise
h
I think I pulled the latest version on friday. Will check tomorrow
@green-football-43791, I'm using acryl-datahub==0.8.23.0 , UI shows 0.8.23 also
Well, if I ingest empty Upstreams, they're cleared, but the corresponding Downstreams are still present. For example, I've got lineage Dataset1 -> Dataset2. If I ingest empty Upstreams, Dataset2 no longer has Dataset1 in Upstreams, but Dataset1 still has Dataset2 in Downstreams.
32 Views
Open in Slack
PreviousNext
Powered by