Datahub UI isn't picking up my entity when I re-in...
# ingestiono
orange-flag-48535
10/03/2022, 3:32 PMDatahub UI isn't picking up my entity when I re-ingest it after a soft delete. The soft-delete sets '{"removed":true}' for the "status" aspectName, as per my look into the Mysql. But the re-ingest isn't touching the removed:true entry at all. My re-ingest is based on sending an UPSERT of with aspectName "schemaMetadata", and the same urn value as the original ingest.
orange-flag-48535
10/03/2022, 4:17 PMThe Hard Delete version of the API works fine (/entities?action=delete) in conjunction with a subsequent UPSERT of schemaMetadata. Is it ok to use hard deletes, or do you have any recommendations about using soft-deletes? For my use case we're fine with permanently wiping out data.
m
mammoth-bear-12532
10/03/2022, 4:26 PM
during a reingest you should re-emit the status aspect with
removed: Falsemammoth-bear-12532
10/03/2022, 4:26 PM
all the out-of-the-box connectors do this
mammoth-bear-12532
10/03/2022, 4:26 PM
I assume you are emitting metadata manually in this case?
o
orange-flag-48535
10/03/2022, 4:30 PMThanks. We're using RestEmitter but only invoking UPSERT with schemaMetadata (and a tableProperties update). There's no use of Status object there. Probably that's the issue! Will see if I can get that part of code modified.
orange-flag-48535
10/04/2022, 5:01 AMEmitting "removed:false" for aspectName "status" along with schemaMetadata etc is making the entry show up in the UI after the soft-delete.
But I see in mysql that there are two entries now for the "status". Is Datahub smart enough to pick up the latest of these two?
mysql> select metadata from metadata_aspect_v2 where urn='urnlidataset:(urnlidataPlatform:myDp,myDb.myTable,DEV)' and aspect='status';
+-------------------+
| metadata |
+-------------------+
| {"removed":false} |
| {"removed":true} |
+-------------------+
s
square-activity-64562
10/04/2022, 11:05 AMYou need to check the version column.
version = 0