hi, I have a custom connector that supports stateful ingestion. and it soft deletes stale metadata as it should. but if I add back the soft-deleted entities (such that they get picked up by the recipes again) they do no get reinstated. is this by design or am I missing something?
edit:
fixed this by using make_dataset_urn_with_platform_instance
rather than make_dataset_urn
when generating urns for dataset entities.
edit2: the above just produced duplicates with slight variation in the urn and I mistakenly thought these were soft-deleted entries being reinstated. the real fix, for anyone reading this in the future, is to add the aspect
Status(removed=False)
to your dataset snapshot object, for example
snapshot = DatasetSnapshot(
urn=dataset_urn,
aspects=[Status(removed=False)],
)