Hi Team, We have set following flush config for r...
# troubleshooting
a
Hi Team, We have set following flush config for realtime table "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "12h", "realtime.segment.flush.threshold.segment.size": "200M", "realtime.segment.flush.autotune.initialRows": "10000000", But still seeing segments getting generated with size less than 50mb (attaching screenshot for reference ) Can someone help?
m
your config looks ok to me. Can you look for logs that have
SegmentSizeBasedFlushThresholdUpdater
(logs/pinot-all.log). Should be in the Controller logs I think
s
These are compressed sizes. Take a look at the size taken up by the segment on your server
a
checked server /pinot/data/server/index/table_REALTIME/table__0__86__20220103T0029Z , directory size is close to what was desired. Thanks for the clarification. @Subbu Subramaniam