In stateful ingestion, do we have any way of forcefully making sure that the source datastore and datahub are same. Right now with hive, even with a successful ingestion many datasets/tables are not getting deleted from datahub. soft_deleted_stale_Entities is []. datahub version : v0.8.38

Can you share gist of your recipe?

@rich-policeman-92383 just confirming, the tables which are not getting deleted are actually deleted from hive or are satisfying your deny patterns?

Those tables have been deleted from hive. I do not see those tables in the ingest logs but they are present in datahub

Were these tables ingested using same same recipe specifically pipeline?

Recently we have changed the sink from rest to Kafka as on rest we were getting random 401

and `pipeline_name`` was preserved ?

Yes pipeline name is the same from the beginning

Ok. this is interesting case. Let me do some research and come back.

Sure, thanks