Hello ! , i am working on managing offline flows (...
# troubleshootingd
Deepak Mishra
08/05/2021, 11:52 AMHello ! , i am working on managing offline flows ( from realtime to offline table) with bucketTimePeriod - 4h with 2M record with bufferTimePeriod - 5d which will move data into offline table daily with maxNumRecordsPerSegment-10M .How can we manage memory in realtime table with 5 days bufferTimePeriod?
m
Mayank
08/05/2021, 2:08 PM
Are you seeing issues with 5d buffer time
d
Deepak Mishra
08/09/2021, 9:57 AM@Mayank , Basically, i am trying to know how much realtime segment data it will store in 5 days . our target is to ~10M daily .exploring performance and tuning apache pinot
Deepak Mishra
08/10/2021, 9:08 AM@Mayank , we are keeping bufferTimePeriod as 5 days and lets assume streaming data will come late 2-3 days and the real time segment went online of this time windows . Does it create new segment for this late data or it will merge with old segment?
m
Mayank
08/10/2021, 1:28 PM
It will become part of the open segment that is currently in consuming state.