Hi. Is there any way to remove the already ingeste...
# ingestiong
gorgeous-glass-57878
06/09/2021, 6:05 PMHi. Is there any way to remove the already ingested datasets in datahub? Or is it an append-only ingestion pipeline? Is there a way to mark 'Deleted' on a particular dataset?
g
gray-shoe-75895
06/09/2021, 6:24 PMDataHub primarily uses “soft deletes” - you can mark a given entity as “removed” by emitted a Status aspect with removed=True
gray-shoe-75895
06/09/2021, 6:26 PMgray-shoe-75895
06/09/2021, 6:30 PMCreating the removal MCE would be something along these lines:
Copy code
mce = MetadataChangeEventClass(
proposedSnapshot=DatasetSnapshotClass(
urn=urn,
aspects=[StatusClass(removed=True)],
)
)f
fancy-fireman-15263
06/09/2021, 6:37 PM@powerful-jelly-19645
👀 1
🙌 1
b
better-orange-49102
06/12/2021, 9:45 AMas i am using file to rest api, this is how my file looks like
Copy code
[{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.DatasetSnapshot": {
"urn": "urn:li:dataset:(urn:li:dataPlatform:kafka,SampleKafkaDataset,PROD)",
"aspects": [
{
"com.linkedin.pegasus2avro.common.Status": {
"removed": true
}
}
]
}
}
}]better-orange-49102
06/12/2021, 9:49 AMinterestingly, putting removed=True removes it from the UI and search results, however, if the dataset is linked upstream/downstream, then it can still be accessed. should this be considered a bug?
g
gray-shoe-75895
06/14/2021, 5:25 PMYep removed=True is essentially a soft delete, which you’d use to indicate that the dataset no longer exists or has been deleted without completely purging the history from the backend storage layer
gray-shoe-75895
06/14/2021, 5:25 PMWe don’t yet support “hard deletes” but are working on it, since there are use cases for both types of deletes
2 Views