Hi Team,
We have set following flush config for realtime table
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "12h",
"realtime.segment.flush.threshold.segment.size": "200M",
"realtime.segment.flush.autotune.initialRows": "10000000",
But still seeing segments getting generated with size less than 50mb (attaching screenshot for reference )
Can someone help?
m
Mark Needham
12/30/2021, 4:54 PM
your config looks ok to me. Can you look for logs that have
SegmentSizeBasedFlushThresholdUpdater
(logs/pinot-all.log). Should be in the Controller logs I think
s
Subbu Subramaniam
12/30/2021, 5:53 PM
These are compressed sizes. Take a look at the size taken up by the segment on your server
a
Anish Nair
01/04/2022, 8:16 AM
checked server /pinot/data/server/index/table_REALTIME/table__0__86__20220103T0029Z , directory size is close to what was desired. Thanks for the clarification. @Subbu Subramaniam