Hi, I noticed some strange behavior when setting

realtime.segment.flush.threshold.rows

for my realtime tables. It seems that the actual number of rows per segment becomes some value smaller than the value I set. For example, I'll set this to 1000000, but in the segment metadata,

segment.flush.threshold.size

would be 500000 and the segment does only ingest 500000 rows. This seems to only happen for some tables, and sometimes it is shrunk by a factor 2 or 4. Just wondering if there is any other setting I'm missing that is causing this?

Can you check if it gets divided by number of partitions?

Oh that might be it. In one example, I have about 20 partitions and it gets divided by 4. Is this expected?

Yes if you have each server consuming 4 partitions

I see, does the number include replicas as well?

It boils down to how many partitions a server has to consume (function of how many partitions there are, how many servers, and what’s the replication)

got it, thanks!

No, what I mentioned above only applies to

realtime.segment.flush.threshold.rows

have added an explanation of what Mayank said above to https://docs.pinot.apache.org/configuration-reference/table#indexing-config