Hi all, I’m trying to wipe down all metadata after dropping mysql tables and deleting elastic indexes I can still see the datasets from ui. I’m running datahub with helm can you assist please?

Hi @better-fireman-33387, you’re probably going to want to do a rollback: https://datahubproject.io/docs/how/delete-metadata/#rollback-ingestion-batch-run

rollback also fails

Hi Nissim, I’m unsure why this might be- could you post the logs from the failed rollback?

The data is only stored in the mysql database and indices (there are several). There is a caching layer in GMS service itself and in the browser. Additionally, there is data in the kafka topics, however that data would only be re-ingested if the zookeeper offsets were reset since the consumer groups would be the same.

How can I list all indices to delete? Rollback is not doing anything and data still remains in ui (partially) The steps I already did to try wipe down all metadata are: 1. Stop gms 2. Delete aspect table 3. Delete elastic indexes 4. Run gms but I can still see some datasets and containera in ui. I wonder if I missed any indexes? I can see not all of them are with datahub prefix Some starts with datajob and some with dataset.

The names of the indices would share the same prefix if setting using

global.elasticsearch.indexPrefix

in the helm chart values for your deployment. Otherwise I would be careful about excluding other indices from being deleted if you are using a shared cluster. The indices for entities are in the form of

<prefix><entity name>index_v2

, two examples would be

containerindex_v2

or

datasetindex_v2

, non-entity indices include

<prefix>graph_service_v1

and

<prefix>system_metadata_service_v1

@brainy-tent-14503 I set the indexPrefix in the chart and then deleted all relevant indices and it worked! Thanks 🙏🏼