Hi team, was wondering if what I’m experiencing is an issue or the expected behavior for a kafka source. It seems like when topics are deleted from kafka cluster, they are not deleted on DH. Is this normal? what do other users do to overcome this situation?

@gray-shoe-75895 is this standard? Any idea what could be going on?

Yup - you can run

datahub delete --platform kafka --entity_type dataset

to clean out the stale metadata, and then enable the stateful_ingestion flag to automatically delete kafka metadata when topics are deleted from the kafka cluster

awesome, will take a look to implementing that