Hi y’all, not sure if I missed any documentation, ...
# troubleshoot
b
Hi y’all, not sure if I missed any documentation, but I’m trying to use
datahub delete
to delete datasets with a specific schema. Looking at https://datahubproject.io/docs/how/delete-metadata/ it looks like it supports a query operator, but I couldn’t figure out exactly how to use it. i.e. I’m trying to do something like
Copy code
datahub delete --enitity_type dataset --env PROD --query "thisschema"
To delete
Copy code
urn:li:dataset:(urn:li:dataPlatform:platform,thisschema.table1,PROD)
urn:li:dataset:(urn:li:dataPlatform:platform,thisschema.table2,PROD)
but NOT
Copy code
urn:li:dataset:(urn:li:dataPlatform:platform,wrong_schema.thisschema,PROD)
The query operator seems to match all 3 since the target string is in the table-name. Is this possible?
📖 1
🔍 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
a
Delete by search should be supported, but I believe we recently added another arg to make it possible
CC: @big-carpet-38439 - do you remember what that arg was?
b
Don’t suppose either of y’all remembered? I don’t have access to the source-code for the cli tool, I assume? 🙂
a
I spoke with the team and this should be supported as of the most recent version- what version are you running?
@brainy-tent-14503 may be able to provide insight here
b
Thanks Paul 🙏, I’ve tried with both the latest datahub cli and the one matching my version (0.9.5).
this should be supported as of the most recent version
This
meaning, it should work with the
query
parameter? I might just not know how to format the argument to that parameter
I assume it just does a blanket urn.contains(query) from the results I’ve seen, but maybe that’s incorrect?
a
I think you’re right- so this may be a bug
b
No worries. Do you know if there is documentation around how to manually delete things? I feel like I always miss a datastore or mess something up whenever I try 😂
a
The cli looks to be using a structured elasticsearch query to find the urns.
Copy code
datahub delete -n --entity_type dataset --soft --query 'fieldPaths:field_foo'
[2023-04-21 18:52:20,036] INFO     {datahub.cli.delete_cli:290} - datahub configured with http://localhost:8080
[2023-04-21 18:52:20,095] INFO     {datahub.cli.delete_cli:326} - Filter matched 1  dataset entities of None. Sample: ['urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)']
  0% (0 of 1) |                                                                                                                                                                         | Elapsed Time: 0:00:00 ETA:  --:--:--[2023-04-21 18:52:20,096] INFO     {datahub.cli.delete_cli:416} - [Dry-run] Would soft-delete urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)
100% (1 of 1) |#########################################################################################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
1 entities with unknown rows will be affected. Took 1.132 seconds to evaluate.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/cli/delete_cli.py#L169
In that example I am looking to delete dataset by column (fieldPaths)
field_foo
the syntax is the same as the search bar when using the
/q
prefixed queries
b
Ooh amazing, thanks David, I didn’t realize the cli was on github, thought it was an Acryl thing 🙂