Hi Team, I'm trying to delete urns and entities i...
# troubleshoot
s
Hi Team, I'm trying to delete urns and entities in our prod datahub environment which has 0.9.2 version. In python, I used the delete cli api which goes like this
Copy code
delete_cli.delete_with_filters(dry_run=False,
                                                                 force=True,
                                                                 include_removed=True,
                                                                 soft=False,
                                                                 entity_type=self.entity_type,
                                                                 platform=self.platform,
                                                                 env=self.env,
                                                                 search_query="*")
however, I encountered a server error
The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.
I'm guessing the query filter returned a lot of entities, but even if I tried to narrow down the search filter it still encounters the same issue. moreover, I also tried to delete the entities using the delete cli through bash and encountered these messages
Discrepancy in entities yielded 0 and num entities 2255. This means all entities may not have been deleted.
can you please advise on how we can address these issues? we're currently in the middle of upgrade and wanted to have a clean slate of entities. thank you in advance
📖 1
🔍 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
s
btw, I switched up the clients being used because I saw in some similar thread that the issue was fixed on 0.9.6. basically I tried 0.9.2, 0.9.6 and 0.10.1 clients but the issues still persists.
d
Hello Ramoncito, thank you for letting me know. @gray-shoe-75895, I came across your thread a few months ago and I'm curious if this issue has been resolved in the latest version? 🙂https://datahubspace.slack.com/archives/C029A3M079U/p1674106690162149?thread_ts=1673271776.412179&cid=C029A3M079U
g
That error message usually shows up when datahub is backed up on processing changes (specifically kafka lag on processing the MetadataChangeLog), and usually resolves itself after a while
w
Hi - we are seeing this issue with Datahub v 0.11, no matter how long we wait, we see this :-
[2023-11-02 11:50:29,812] WARNING  {datahub.cli.cli_utils:439} - Discrepancy in entities yielded 0 and num entities 927. This means all entities may not have been deleted.
g
What is the delete command that you're trying to run?
w
Copy code
datahub delete --platform s3 --hard
for example - if there are >10,000 entities to be deleted, then things go awry and this discrepancy appears (with differing numbers) - we can rerun the command , but at the end we end up with this message - see also https://github.com/datahub-project/datahub/issues/9171
g
If you try with the latest datahub cli (v0.12.0.3), does this still happen?