https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
i
check the overlord log for the taskid. It may have more information
o
Hi Ian, nothing in the Overlord log for this task...
i
hmmm, I would think there would be. Try grep with the ID in the OL log as well as the MM log.
Does the console show a reason for the failure if you click on the task?
o
Sorry my bad... I grep on the wrong ID... I have that in the OL:
Copy code
2023-03-30T13:13:36,603 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] because: [Task is not in knownTaskIds]
2023-03-30T13:13:36,605 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: 6.1.245.106:8091, status 200 OK, response: {"task":"index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc"}
2023-03-30T13:13:36,705 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] because: [Task is not in knownTaskIds]
2023-03-30T13:13:36,707 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: 6.1.245.106:8091, status 200 OK, response: {"task":"index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc"}
2023-03-30T13:13:42,519 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[6.1.245.106:8091] wrote FAILED status for task [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] on [TaskLocation{host='6.1.245.106', port=8101, tlsPort=-1}]
2023-03-30T13:13:42,519 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[6.1.245.106:8091] completed task[index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] with status[FAILED]
2023-03-30T13:13:42,519 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc
2023-03-30T13:13:42,520 ERROR [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Ignoring notification for already-complete task: {class=org.apache.druid.indexing.overlord.TaskQueue, task=index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc}
2023-03-30T13:13:42,521 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] because: [notified status change from task]
2023-03-30T13:13:42,521 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] on worker[6.1.245.106:8091]
2023-03-30T13:13:42,523 WARN [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Unknown task completed: index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc
2023-03-30T13:13:42,523 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task FAILED: index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc (1241680 run duration)
2023-03-30T13:13:42,523 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task[index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] went bye bye.
Shutdown [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] because: [Task is not in knownTaskIds]
can be the cause ?
i
Yes, that is the cause. It looks like the overlord did not know of the running task and tried to issue a shutdown. The task had already succeeded so I think it is ok that it was just marked failed by the overlord
Did the overlord restart during this task running or anything like that?
o
The task had already succeeded
actually the task ends due to this shutdown (after only 20minutes running)... and no , no restart
Why the OL wouldn't know the task ? not registered in Metadata/Zookeeper ? Don't know who handle that
i
The overall reason for this is the overlord instructed the task to be killed due to an issue (unknown task) and the task reports success in executing the kill and doing what it was instructed to do.
I see it mentions curator in the threads so I am guessing it was not found in zookeeper
the overlord did what it should in that it does not want rogue tasks running
o
OK the list of running tasks is stored in ZooK ?
i
I believe so
there is a config that controls this you may want to explore -> https://druid.apache.org/docs/latest/configuration/index.html#overlord-operations 
druid.indexer.runner.type
o
ok so default is 
httpRemote
without ZK
i
that is the latest version, not sure what version you are on so it may not be set to that
o
ok let me check
We're in 24.0 so I guess it is 
httpRemote
. How can I check that directly in the system ?
i
you can check in runtime.properties of overlord
o
damn...
Copy code
druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata
Ok its only by dafault in 25.0 https://github.com/apache/druid/pull/13092
Thanks @Ian Roberts, will try to use httpRemote and see if its works better...
i
You are welcome, good luck!
4 Views
Open in Slack
PreviousNext
Powered by