best-eve-12546
04/11/2023, 9:24 PMdatahub delete to delete datasets with a specific schema. Looking at https://datahubproject.io/docs/how/delete-metadata/ it looks like it supports a query operator, but I couldn’t figure out exactly how to use it.
i.e. I’m trying to do something like
datahub delete --enitity_type dataset --env PROD --query "thisschema"
To delete
urn:li:dataset:(urn:li:dataPlatform:platform,thisschema.table1,PROD)
urn:li:dataset:(urn:li:dataPlatform:platform,thisschema.table2,PROD)
but NOT
urn:li:dataset:(urn:li:dataPlatform:platform,wrong_schema.thisschema,PROD)
The query operator seems to match all 3 since the target string is in the table-name.
Is this possible?lively-cat-88289
04/11/2023, 9:24 PMastonishing-answer-96712
04/12/2023, 7:58 PMastonishing-answer-96712
04/12/2023, 7:59 PMbest-eve-12546
04/17/2023, 8:55 PMastonishing-answer-96712
04/18/2023, 4:31 PMastonishing-answer-96712
04/18/2023, 4:31 PMbest-eve-12546
04/18/2023, 4:33 PMthis should be supported as of the most recent version
This meaning, it should work with the query parameter? I might just not know how to format the argument to that parameterbest-eve-12546
04/18/2023, 4:33 PMastonishing-answer-96712
04/20/2023, 8:48 PMbest-eve-12546
04/21/2023, 4:12 PMaloof-gpu-11378
04/21/2023, 11:52 PMaloof-gpu-11378
04/21/2023, 11:53 PMdatahub delete -n --entity_type dataset --soft --query 'fieldPaths:field_foo'
[2023-04-21 18:52:20,036] INFO {datahub.cli.delete_cli:290} - datahub configured with http://localhost:8080
[2023-04-21 18:52:20,095] INFO {datahub.cli.delete_cli:326} - Filter matched 1 dataset entities of None. Sample: ['urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)']
0% (0 of 1) | | Elapsed Time: 0:00:00 ETA: --:--:--[2023-04-21 18:52:20,096] INFO {datahub.cli.delete_cli:416} - [Dry-run] Would soft-delete urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)
100% (1 of 1) |#########################################################################################################################################################################| Elapsed Time: 0:00:00 Time: 0:00:00
1 entities with unknown rows will be affected. Took 1.132 seconds to evaluate.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/cli/delete_cli.py#L169aloof-gpu-11378
04/21/2023, 11:53 PMfield_fooaloof-gpu-11378
04/21/2023, 11:54 PM/q prefixed queriesbest-eve-12546
04/24/2023, 1:52 PM