Hello ! , i am working on managing offline flows (...
# troubleshooting
d
Hello ! , i am working on managing offline flows ( from realtime to offline table) with bucketTimePeriod - 4h with 2M record with bufferTimePeriod - 5d which will move data into offline table daily with maxNumRecordsPerSegment-10M .How can we manage memory in realtime table with 5 days bufferTimePeriod?
m
Are you seeing issues with 5d buffer time
d
@Mayank , Basically, i am trying to know how much realtime segment data it will store in 5 days . our target is to ~10M daily .exploring performance and tuning apache pinot
@Mayank , we are keeping bufferTimePeriod as 5 days and lets assume streaming data will come late 2-3 days and the real time segment went online of this time windows . Does it create new segment for this late data or it will merge with old segment?
m
It will become part of the open segment that is currently in consuming state.