Hi all! We hard-deleted a dataset. It cannot be fo...
# all-things-deploymentb
billions-rose-75566
07/31/2023, 8:44 AMHi all! We hard-deleted a dataset. It cannot be found anymore in the UI, except for the recently viewed... Is this normal behaviour ?
m
many-glass-1784
07/31/2023, 9:30 AMThis feels similar to an issue I had a while ago where I did something similar. I also did a hard delete and the item was gone from the Datahub database, but also still showed up in the recently viewed/recommendations/... . After looking around I found that those items seem to come from ElasticSearch and not from the database and a hard delete doesn't seem to remove the corresponding items in ElasticSearch. I don't know if is a bug or a feature.
b
billions-rose-75566
07/31/2023, 9:32 AMThanks for sharing this! Maybe a follow up question, but i want to doublecheck that a GraphQL query to retrieve (existing) datasets does not include the hard-deleted ones even though they come from ElasticSearch. Can someone verify that?
billions-rose-75566
07/31/2023, 9:32 AMOtherwise it does not have a lot of value to just do a hard-delete in the Datahub tables. I would manually have to delete things too in Elasticsearch...
m
many-glass-1784
07/31/2023, 9:35 AMThe GraphQL stuff specific to datasets worked correctly for me and only returned existing datasets (so without hard deleted ones).
many-glass-1784
07/31/2023, 9:37 AMb
billions-rose-75566
07/31/2023, 9:37 AMAh okay thanks. Hopefully Datahub will either fix it or it is an intended choice to do that. Either way, it can be a bit confusing, since the "recently viewed" also comes from a GraphQL call. To really understand it i need to dive deep in the code which is not my preference đ
m
many-glass-1784
07/31/2023, 9:39 AMThey seem to use different ElasticSearch indexes and one of those indexes isn't kept in sync with hard deletes it looks like to me. So from my side it looks like a bug.
b
billions-rose-75566
07/31/2023, 9:40 AMIndeed my thoughts as well. Hopefully someone from Acryl can think with us.
billions-rose-75566
08/02/2023, 6:52 AM@delightful-ram-75848 sorry for mentioning you directly. Perhaps you or another colleague from Acryl could take a look at our issue? I still don't know if this is a bug or intentional behaviour... đ
a
aloof-gpu-11378
08/02/2023, 6:54 AMThanks for the head up - @gray-shoe-75895 Could you check if this expected behavior?
g
gray-shoe-75895
08/04/2023, 3:28 AMDeleted stuff showing up in ârecently viewedâ sounds like a bug - definitely not expected behavior
gray-shoe-75895
08/04/2023, 3:30 AMJanâs diagnosis here sounds correct - iirc we generate the recently viewed section from datahub_usage_events, whereas the hard deletes impact mysql and the <entity_type>_* indexes
gray-shoe-75895
08/04/2023, 3:31 AMcc @dazzling-yak-93039 - looks like we need an extra check to hide deleted entities from the ârecently viewedâ section
b
billions-rose-75566
08/04/2023, 7:56 AMThanks @gray-shoe-75895 for looking into this! Perhaps im asking too early, but when can we expect this fix for the upcoming release?
a
aloof-gpu-11378
08/07/2023, 6:26 PMI created a ticket for this internally! We will prioritize it and get to it when we can!
s
stocky-monkey-23735
08/22/2023, 6:41 AMIs there a way to do this manually while the fix is being developed? To clean up what's on the UI.
m
modern-orange-37660
12/13/2023, 3:11 AMIs there any progress in this story? Facing the same problem
l
lively-iron-79328
01/24/2024, 9:57 AMHi, I'm having the same issue, is there any update regarding this issue? Thank you!
m
many-glass-1784
01/24/2024, 10:08 AM@lively-iron-79328 I haven't gotten any updates from @astonishing-answer-96712 or @dazzling-yak-93039 (which he referenced in my original thread) about this issue and as far as I know it hasn't been fixed.
a
ambitious-bird-91607
01/24/2024, 11:19 AMFacing same problem here