What is the purpose of system_metadata_service_v1 index in Elasticsearch? I see a lot of ingestion events present there (e.g. datasets)?

The

system_metadata

index is used for performing rollbacks of ingestion. There might be a few more use-cases for that index that have cropped up recently, but that is the main use for it.

Thanks @mammoth-bear-12532! Initially it got me worried as I was wondering if it was serving as a catch-all or dead letter queue kind of index. It is one of our largest indices with individual entity level details.. I am wondering if there is a retention set to these docs so they get flushed out periodically (when a rollback may not be required?) or is there a maintenance job that runs(or needs to run) to keep this from exploding..

We are working on a design for this. Will update this thread when we have something to share :)