https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
a
Hi! Could you please check if segments are not getting published or if the handoff is unsuccessful from the task logs? If there is a log like 
Waiting for handoff
at the end of the task logs, could you please check the coordinator/time metric and historical availability as well?
👍 1
e
Oh I found tons of 
Still waiting for handoff for [%d] segments
seems like peon tasks were successful but publishing failed also, I found 
Shutdown [index_kafka_xxx] because: [task is not in knownTaskIds...
a
could you please check the coordinator/time metric and historical availability as well?
It would be great if you could share which duty is taking the most time in case the coordinator cycles are long
👍 1
t
Hello, try increasing 
completionTimeout
in your supervisor spec
By default, 
taskDuration
= 
completionTimeout
. In my case, 
taskDuration
was 1800 seconds. I’ve increased 
completionTimeout
to 3600 seconds, and issue is no more. I’ve got this issue when lag is growing. When there is a lot of Kafka data to ingest, it seems the 
taskDuration
is not long enough to publish segments and tasks failed with a long duration
a
It seems like the issue here is that segments have been published but not handed off based on the logs
t
I think the main issue here is : 
"No task in the corresponding pending completion taskGroup[group_id] succeeded before completion timeout elapsed"
And I’ve managed to get rid of it doing what I’ve suggested. I think it’s worth a shot
Yup, I must edit my previous message, but it seems I can’t
By default, 
taskDuration
= 
completionTimeout
. In my case, 
taskDuration
was 1800 seconds. I’ve increased 
completionTimeout
to 3600 seconds, and issue is no more. I’ve got this issue when lag is growing. When there is a lot of Kafka data to ingest, it seems the 
taskDuration
is not long enough and tasks failed with a long duration
👍 1
e
Thanks for reply friends. I once changed 
druid.segmentCache.locations=[{"path":"/data1/druid/segment-cache/","maxSize":"6200g"}]
to 
druid.segmentCache.locations=[{"path":"/data1/druid/segment-cache/","maxSize":"4800g", "freeSpacePercent":20..0}
] and as Amatya said, this could be problem. I’ll consider what you said. thansk!
Hey I found that 
completionTimeout
should be quite long in some cases. I’m running big cluster and consumes terabytes of data in a day. When kafka lag is huge, some tasks take more than 2 hours (
taskDuration
=1hr) and even 
PT3600S
wasn’t enough. I gave 
PT7200S
and Druid could manage tasks to be completed. After few hours, tasks took about 1HR again. I twaeaked some other options so I’m not sure this is the only solution. but it did work anyway 😅 Thank you !
t
Hello Eddie, yep this is what I’ve experienced too. Once the lag is caught up, everything goes back to normal. Glad to hear you solved it
👍 1
2 Views
Open in Slack
PreviousNext
Powered by