I am trying to debug a strange issue regarding segment generation from a realtime table. Its config is set up like this:

"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.segment.size": "500M",
"realtime.segment.flush.threshold.time": "4h"

However, when inspecting the metadata of any of the realtime segments we can see for example:

"segment.realtime.endOffset": "67399447",
"segment.start.time": "1762424217000",
"segment.time.unit": "MILLISECONDS",
"segment.flush.threshold.size": "100000",
"segment.realtime.startOffset": "66512835",
"segment.size.in.bytes": "14018213",  <====== 14MB instead of 500M    
"segment.end.time": "1762426143000",  <====== subtracting segment.start.time from this we get roughly 35 min 
"segment.total.docs": "100000",
"segment.realtime.numReplicas": "1",
"segment.creation.time": "1762511599197",
"segment.index.version": "v3",
"segment.crc": "3704033136",
"segment.realtime.status": "DONE",

it’s a bit buried in blog posts, but the first segment size starts with 100k rows and gradually approaches the threshold values in your config as Pinot understands the shape of your data, https://docs.startree.ai/recipes/configuring-segment-threshold#desired-segment-size

You are amazing. Thank you, it would have been hard to come by this info