This message was deleted.
# troubleshootings
Slackbot
01/12/2023, 7:06 AMThis message was deleted.
i
Igor Berman
01/12/2023, 7:14 AMthis is what I see in log of the task itself
it finishes successfully, there is some error connected to lookups
but probably connected to shutdown sequence
v
Vijay Narayanan
01/12/2023, 7:43 AMI have seen this when there was a firewall on the machine running the task (the firewall was left enabled on one node out of many and it took a while to track that down). Does this happen on a specific node?
i
Igor Berman
01/12/2023, 9:30 AMthanks @Vijay Narayanan I will check once again firewall settings. maybe it’s some sporadical flips of some rules…not sure. It’s not happening all the time but only once in a while.
Let me understand this better: you are saying that this kind of problems might happen because of some connectivity issues between node executing peon task(middle manager) and coordinator?
v
Vijay Narayanan
01/12/2023, 11:53 AMyes. If the task is not able to communicate with the overlord.
i
Igor Berman
01/12/2023, 1:29 PMok, thanks. Are there any configs for retries/backoff policy etc?
Igor Berman
01/12/2023, 1:39 PM@Vijay Narayanan hmm,
looking at peon logs once again I see
2023-01-12T022051,924 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_sp_campaigns_realtime_aggregation_cdce20b447c1666_ghjeoknb].
2023-01-12T022051,924 INFO [main] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping gracefully (status: [PUBLISHING])
2023-01-12T022051,924 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Graceful shutdown of task[index_kafka_sp_campaigns_realtime_aggregation_cdce20b447c1666_ghjeoknb] finished in 0ms.
For me it seems like the part of talking with overload finished successfully, what I”m missing?
v
Vijay Narayanan
01/12/2023, 1:40 PMcan you paste the full overlord log here?
Vijay Narayanan
01/12/2023, 1:41 PMis this happening only with tasks on a specific node?
i
Igor Berman
01/12/2023, 1:43 PMit’s happening from time to time from all middle manager’s nodes
v
Vijay Narayanan
01/12/2023, 1:43 PMlooking at the log I am seeing rg.apache.druid.indexing.overlord.TaskQueue - Failed to persist status for task…..do you see any metastore errors in the rest of the log
i
Igor Berman
01/12/2023, 1:43 PMchecking
Igor Berman
01/12/2023, 1:44 PMyes, I see 4 such errors today
Igor Berman
01/12/2023, 1:44 PM“apache.druid.indexing.overlord.TaskQueue - Failed to persist status for task”
Igor Berman
01/12/2023, 1:45 PMlooking at previous dates, it’s always 2-4 such errors per day
v
Vijay Narayanan
01/12/2023, 1:46 PMif the overlord leader has changed after the task started then this may happen….is the overlord restarting frequently?
i
Igor Berman
01/12/2023, 1:47 PMI’ll check this
Igor Berman
01/12/2023, 1:47 PMthanks
v
Vijay Narayanan
01/12/2023, 1:47 PMif you have a lot of tasks check the heap size on the overlord and consider increasing it.
i
Igor Berman
01/12/2023, 1:53 PMany other options? I’m looking at heap usage, and it seems normal, gc is low
the process running from 19Dec, so it’s not restarted recently
v
Vijay Narayanan
01/12/2023, 2:03 PMThe other possibility is an issue with zookeeper and the communication with zookeeper. Which version of druid is this?
i
Igor Berman
01/12/2023, 2:35 PMWe are using 24.0.1.
I will check zookeeper metrics, might be something there
thanks a lot!
Igor Berman
01/12/2023, 5:57 PMit seems to me that this one is very close to what we are experiencing https://www.druidforum.org/t/kafka-ingestion-peon-tasks-success-but-overlord-shows-failure/7374/28
I’ll try to check options and relevant metrics