Hi Team, I am looking for some guidance on recovering the backlog observed in minion R2Off task. --> current for one of our customer we have a hybrid table with retention of 20 days and R2OFF schedule every 15 min bucket time we changed to 1h (from 2h) and buffer time is 2d. --> retention we increased to 25 days from 20 days to avoid data drop. --> minion task config are 6hr timeout and 2 tasks in parallel. table config in RT table keeps segments sized based on 500MB or 2hr whichever is first. "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "schedule": "0 */15 * * * ?", "bucketTimePeriod": "1h", "bufferTimePeriod": "2d" } no of minions configured today are 15. and resources per minion are as below - resources: limits: cpu: '5' memory: 18Gi requests: cpu: '4' memory: 16Gi i would like to understand how this parallel tasks get executed. because event after making it 2 i see only 1 task per table getting created at any point in time. what configurations i can make to get the backlog reduced to 2 or 3 days and process backlogs faster.

The

bucketTimePeriod

maybe the bottle neck as there might not be enough segments getting picked per bucket to generate enough subtasks.

thanks for the response shounak their are tasks running for close to 4+ hours. i can confirm that their are many segments available. reason to say this is watermark today for a table is between 4 and 6 Feb and we are on 26th feb their is enough lag to process. i believe this means many segments to move.

Although there are too many segments pending only the segments that overlap the processing window will be picked at once ref : https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows

when we say majority time spent we mean from downloading segment, converting to offline and then uploading back to s3 interface right. i assume this info is seen either in logs or in Grafana dashboard can u give me some inputs on this part. another question i have if what u mention is the case i hardly have 5-6 RT2OFF tables so having 15 minion pods does that make sense?

i changed it to 1hr from 2hr when i saw many segments being processed in every task

what is the typical range of data each segment has in the realtime table?

we configure it to be 500MB size per say and 2hr in Realtime segment config so 500MB or 2hr whichever is early. hope i understood ur question

oh sorry you mentioned that earlier, missed it. Yeah so its beneficial to have 2hr+ bucket else the segments would be picked twice to process (one for each 1 hr bucket)

no problem. ok i can get the bucket back to 2hr. if you can kindly let me know why inc no of parallel task to 2 does not work. i believe u mentioned it depends on no of buckets to process based on bucket config so will inc bucket to 4hr from 2hr will help or i am not thinking right here

instead of having 15 minions you can maybe benefit from having a single bulky minion

yes we are already doing that i dont like 1 bulky minion option 🙂 as all table tasks willl compete their

Oh I meant 1 for each lagging table, also reduce the numConcurrentTaskPerInstance to 1 as 2 tasks may end up in contention of resource for execution

ok i shall roll back no of tasks as we started seeing timeout again after inc from 1 to 2

One thing you can do is parallelise the segment download by setting higher value for

segmentDownloadParallelism

in task configs

ok thanks i shall check this