https://pinot.apache.org/ logo
Join Slack
Powered by
Our RealtimeToOfflineSegmentsTask seems to be stuc...
# troubleshooting
s
Our RealtimeToOfflineSegmentsTask seems to be stuck and I'm having a hard time diagnosing why. The controller repeatedly reports: Found incomplete tasks: [Task_RealtimeToOfflineSegmentsTask_1659285958877] for same table: immutable_events_REALTIME. Skipping task generation.
m
@Xiaobing ^^
s
The minion reports this and nothing else
Initialized mapper with 31 record readers, output dir: /var/pinot/minion/data/RealtimeToOfflineSegmentsTask/tmp-3533971d-9eb6-4682-9d66-a9fa480b22ce/workingDir/mapper_output, timeHandler: class org.apache.pinot.core.segment.processing.timehandler.EpochTimeHandler, partitioners:
m
Anything in the minion logs
s
That's it above, it got to that point and is logging nothing else now
I'm going to increase my minion pvc and restart it and see
ok did that and minion is reporting that it's downloading segments, copy, fetched...etc.
So I'll keep an eye on that and see, could have been a space issue
👍 1
I bet you that was it, but it's weird the minion just stopped altogether and reporting no space issue. It's loading segments now so stay tuned.
well now it's back to that same spot and sitting there, before when it got there I let it run for about and hour and nothing
The minion is at this point:
Initialized mapper with 31 record readers, output dir: /var/pinot/minion/data/RealtimeToOfflineSegmentsTask/tmp-1900932a-05f1-4352-92b3-b78a8d705161/workingDir/mapper_output, timeHandler: class org.apache.pinot.core.segment.processing.timehandler.EpochTimeHandler, partitioners:
and controller is repeatedly saying: Found incomplete tasks: [Task_RealtimeToOfflineSegmentsTask_1659285958877] for same table: immutable_events_REALTIME. Skipping task generation.
So I will give it a bit and see.
m
We might need to cleanup those tasks, let me get back
s
ok, I tried to delete the tasks and run the cleanup as well.
k
Can you list the disk usage.. it’s possible that tasks are failing but temp files are not getting deleted
s
image.png
image.png
looks like I have tmp dirs from a couple of tries, but still plenty of space left it seems
the latset one has a workgDir w/ 3 directories, but all are empty
Minion hasn't logged anything since the last one I pasted
ok it's logging some additional information
2022-07-31T181951.607493428Z Beginning reduce phase on partitions: [0] 2022-07-31T181951.609899257Z Start reducing on partition: 0 2022-07-31T181951.618859966Z Start sorting on numRows: 40852147, numSortFields: 44 2022-07-31T183021.661437510Z Beginning reduce phase on partitions: [0] 2022-07-31T183021.661511283Z Start reducing on partition: 0 2022-07-31T183021.663075552Z Start sorting on numRows: 40852147, numSortFields: 44
it kind of feels like it's running the same job twice, like I didn't clean something up properly.
Is it ok to delete any running jobs and then nuke the tmp- folders if things don't progress?
looks like it's progressing, we had a few days backlog of segments so I just needed to be more patient as it did it's thing I think.
m
Thanks for confirming
s
We are working to get our monitoring dashboards setup so we shouldn't be in this big backlog situation in the future.
👍 1
Task: Task_RealtimeToOfflineSegmentsTask_1659289467700_0 got cancelled _org.apache.pinot.minion.exception.TaskCancelledException_: RealtimeToOfflineSegmentsTask on table: immutable_events_REALTIME, segments: immutable_events__0__136__20220728T2343Z,immutable_events__0__146__20220729T1211Z,immutable_events__0__147__20220729T1325Z,immutable_events__0__152__20220729T1427Z,immutable_events__0__153__20220729T1435Z,immutable_events__0__154__20220729T1443Z,immutable_events__0__155__20220729T1452Z,immutable_events__0__156__20220729T1500Z,immutable_events__0__157__20220729T1508Z,immutable_events__0__158__20220729T1517Z,immutable_events__0__159__20220729T1525Z,immutable_events__0__166__20220729T1623Z,immutable_events__0__170__20220729T1753Z,immutable_events__1__102__20220728T2119Z,immutable_events__1__106__20220729T1010Z,immutable_events__1__109__20220729T1430Z,immutable_events__1__110__20220729T1452Z,immutable_events__1__111__20220729T1514Z,immutable_events__1__114__20220729T1622Z,immutable_events__1__115__20220729T1648Z,immutable_events__2__101__20220729T1430Z,immutable_events__2__102__20220729T1439Z,immutable_events__2__103__20220729T1447Z,immutable_events__2__104__20220729T1456Z,immutable_events__2__105__20220729T1504Z,immutable_events__2__106__20220729T1512Z,immutable_events__2__108__20220729T1529Z,immutable_events__2__115__20220729T1627Z,immutable_events__2__118__20220729T1711Z,immutable_events__2__84__20220728T2303Z,immutable_events__2__95__20220729T1243Z got cancelled at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:195) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f] at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:60) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f] at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:111) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f] at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:88) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f] at org.apache.helix.task.TaskRunner.run(TaskRunner.java:71) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Task: Task_RealtimeToOfflineSegmentsTask_1659289467700_0 completed in: 5077131ms
says the task was cancelled
ran for 80 minutes, seemed to be doing good things, built segments, etc. then reported cancelled
task states right after it reported cancelled:
Copy code
{
  "Task_RealtimeToOfflineSegmentsTask_1659302625239": "IN_PROGRESS",
  "Task_RealtimeToOfflineSegmentsTask_1659289467700": "COMPLETED"
}
handling message: 3bb81900-0506-474c-b8c1-f88b93ef29b1 transit TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1659289467700.TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_1659289467700_0|[] from:RUNNING to:TIMED_OUT, relayedFrom: null
x
to:TIMED_OUT
must be stopped due to timeout the task has 1hr timeout iirc, and it can be increased via a config (finding it in docs)
s
Thanks will check that out!
x
Copy code
Using "POST /cluster/configs" API on CLUSTER tab in Swagger, with this payload
{
	"RealtimeToOfflineSegmentsTask.timeoutMs": "600000", <--- can tweak this value
	"RealtimeToOfflineSegmentsTask.numConcurrentTasksPerInstance": "4" <--- this is a bit misleading, as RT2OFF task is single-threaded for each table. this knob might be helpful when you have RT2OFF tasks from many tables.
}
s
ok, tweaked and we shall see what happens, thanks again!
👌 1
I'm kind of glad we ran into this one though, this big backlog of rollup. Good learnings.
Looks like we have success and the timeout was the issue.
🙏 2
m
Thanks @Stuart Millholland. @Xiaobing can we add this to documentation and/or faq?
👌 1
x
s
Excellent thanks!
m
Thanks @Xiaobing
Open in Slack
PreviousNext
Powered by