Hi team, when using upsert with realtime table, can we do segments compaction or merge for committed segments? Like merge multiple small segments into one large segment. If not, how should we deal with too many small segments when using upsert? @User @User

upsert in pinot does not use compact, but use metadata to track the records of the same key. you can find the design details in this doc https://docs.google.com/document/d/1qljEMndPMxbbKtjlVn9mn2toz7Qrk0TGQsHLfI--7h8/edit

I see, so in the current design, there is no way to merge multiple small segments into larger segments for upsert table? We can only control segments size during ingestion?

that’s right. take a look at these configs https://docs.pinot.apache.org/basics/getting-started/pushing-your-streaming-data-to-pinot

These are two separate topics. Segment merge is not supported yet, and the feature is on our roadmap