Hi, I've noticed my RealtimeToOfflineSegmentTask i...
# generald
Dan DC
02/28/2022, 12:37 PMHi, I've noticed my RealtimeToOfflineSegmentTask is not working anymore. I'm losing segments because they are not moved to the offline table. I only see 2 errors in the logs: one says "Job TaskQueue_RealtimeToOfflineSegmentsTask_Task_RealtimeToOfflineSegmentsTask_.... exists in JobDAG but JobConfig is missing! Job might have been deleted manually from the JobQueue: TaskQueue_RealtimeToOfflineSegmentsTask, or left in the DAG due to failed clean-up attempt from last purge" the other error is specific to a table and says "Got unexpected instance state map: {<list of pinot servers here>} for segment: <segment name here>"
Dan DC
02/28/2022, 12:39 PMCan someone help me to understand the problem? Is there any way to get more information about what's failed? How can is closely monitor the minion tasks to know if they've been successful?
m
Mayank
02/28/2022, 4:36 PM
@User ^^
x
Xiaobing
02/28/2022, 5:00 PMThis seems new to me. Will take a look and reply here.
As to minion observability, I think it emits metrics about task status and there are a few rest endpoints to inspect task status too. But this area needs some improvements for ease of use, which I’m looking into as well.
Xiaobing
02/28/2022, 7:53 PMThe log msg “… but JobConfig is missing!…” is actually from helix code base. Could you share more logs around this log line, including the class name and code line numbers, so I could locate what Pinot code led to it.
btw, what’s the ingestion rate for your RT table? The RT2OFF minion task is a single-threaded task, so it might not catch up with very high ingestion rate. The task watermark under ‘ZK/PropertyStore/Minion_Task_Metadata/’ might help tell the progress.
This log msg “Got unexpected instance state map…” is from code here, when validating the LLC consumers for the RT table. I don’t think it’d impact the minion task directly. But it seems concerning, as it might affect the consumption.
d
Dan DC
03/02/2022, 3:56 PMThanks @User I'm afraid I can't provide lines numbers in the logs but I'll provide the class names. I'll upload a file in a min, the errors are visible in the controller, I can't find anything in minion logs