This message was deleted.

we are using version 24.0.1

A common reason for handoff to fail is if your historicals are full. You could see that on the Services tab of the console

thanks @Vadim, just to clarify you mean full in a sense of disk space configured? we further investigated the problem and found it’s connected somehow to coordinator(we running it with overlord) Restart of leader coordinator helped to release handoffs. As some metric that helped us is that number of jetty connections started to grow on coordinator(probably all other processes tried to call some endpoints of coordinator). The question what metric I can check to understand what happened to coordinator. Looking in logs of coordinator didn’t revealed something special Gc/Heap usage wasn’t high (same as before problem happened)

It sounds like your coordinator is taking a long time in one of it's cycles which means it's not handing off segments quick enough. Do you have metrics enabled (https://druid.apache.org/docs/latest/operations/metrics.html#jetty)? There is a metric for coordinator/global/time which will have a dimension called dutyGroup which will tell you how long per cycle the coordinator is spending on each duty. Hopefully it should be obvious once looking at those metrics to see which duty is problematic...

Hi @Nick M yes we have coordinator/global/time, but it’s reported through prometheus emmiter(so there are some differences) I haven’t noticed something special in coordinator global times(what is normal here?) I don’t have dimension by duty(fixing it now) took

max by (hostname)  (histogram_quantile(1, druid_coordinator_global_time_bucket{job="druid-spd", hostgroup=~".*$hostgroup"}))

and got something like that when green line stopped and yellow line started - is the switch between coordinators due to restart. So if I understand it correctly at problem time, coordinator cycle took up to 10seconds(which I believe not that big)