Hi everyone! When using the realtime table with upsert, is there any compaction mechanism on segments? Or will they just keep on being created and kept forever? Thanks!

Good question, we have discussed it and will likely decide to do it, no concrete plan immediately that I am aware of though.

Thanks for the reply! I was under the impression that on some design document it was mentioned that there would be some background task that would purge the segments of stale entries

Yes, currently that purge job is there for GDPR, and uses the Pinot Minion framework. We need to create on that purges for stale entries