Is there a way to edit what was previously ingeste...
# ingestion
h
Is there a way to edit what was previously ingested? IE if we ran ingestion and then at a later point a table was deleted/deprecated and no longer exists on our data warehouse....what is the appropriate was to remove this entity?
g
DataHub primarily relies on “soft deletes” since we want to retain history - you can do it by adding a “Status” aspect with deleted=True
h
gotcha thanks - and probably a base question but where do I set the deleted flag? I looked in mysql db but didn't see anything there
g
Right now, you need to emit a status aspect for your entity with removed set to true
h
Thanks Gabe - do you know how (or have a link) to the method to make a request to change this field? Or how to update from the container/etc. I'm assuming a rest api call or http....but didn't see an example
g
if you know the set of entities you want to delete, you can use a file source
and create some snapshots that contain the Status aspect assigned to the urn in question
if there were a way to programmatically identify when these deletes happen you could emit an MCE at that point in time.
For example, if you deleted them via an airflow job or some other script, you could emit an mce at that point that marks the entity as deleted
h
Thank you!