https://linen.dev logo
Join Slack
Powered by
We're using Zookeeperless Druid on Kubernetes and ...
# troubleshooting
j
We're using Zookeeperless Druid on Kubernetes and have noticed some issues when the Coordinator restarts: 1. We lose all indexer logs. They're still in deep storage but they're no longer available from the GUI 2. Supervisors get restarted. Unfortunately that 2nd issue causes some big problems for us because having all Supervisors start at the same time causes really bad traffic patterns (over 100 peons all writing to Deep Storage at exactly the same while another 100+ peons all get scheduled to start while the first 100 finish causing cpu and memory spikes within the cluster). I've tried giving a longer grace period (5 minutes) but I don't think this is an unclean shutdown situation.
This log line looks related:
Copy code
2026-09-30T20:55:31,627 ERROR [LeaderElectorAsyncWrapper] org.apache.druid.k8s.discovery.K8sDruidLeaderSelector - listener.stopBeingLeader() failed. Unable to stopBeingLeader: {exception
Type=java.util.concurrent.RejectedExecutionException, exceptionMessage=Task org.apache.druid.server.coordinator.loading.HttpLoadQueuePeon$$Lambda$6516/0x00007f448529a970@5641f1f0 rejecte
d from java.util.concurrent.ThreadPoolExecutor@3a4a007[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10817], class=org.apache.druid.k8s.discovery.K8s
DruidLeaderSelector}
I see this strange log line as well:
Copy code
2026-09-30T20:38:56,983 ERROR [Thread-58] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Failed to unannounce DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceNam
e='druid/coordinator', host='10.251.123.245', bindOnHost=false, port=-1, plaintextPort=8081, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false, labels=null}, nodeRole='COORDINATO
R', services={}', startTime=2026-09-29T16:03:42.069Z, availableProcessors=16, totalMemory=34359738368}]
<http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Failed to patch pod[data/druid-druid-data-coordinator-1], code[422], error[{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "the server rejected our request due to an error in our request",
  "reason": "Invalid",
  "details": {},
  "code": 422
}].
I was able to (somewhat) watch it unfold in realtime... I see 
coordinator-1
relinquish leadership
Copy code
2026-09-30T20:38:57,016 INFO [Thread-58] org.apache.druid.indexing.overlord.DruidOverlord - I am no longer the leader...
Then 
coordinator-1
goes completely offline after 
2026-09-30T20:39:22,881
However, 
coordinator-0
doesn't pick up leadership until later:
Copy code
2026-09-30T20:40:27,615 INFO [LeaderElectorAsyncWrapper] io.kubernetes.client.extended.leaderelection.LeaderElector - Successfully acquired lease, became leader
Copy code
2026-09-30T20:40:28,126 INFO [leader-elector-hook-worker-1] org.apache.druid.k8s.overlord.KubernetesTaskRunner - Locating [293] active tasks.
2026-09-30T20:40:32,610 INFO [LeaderElectorAsyncWrapper] io.kubernetes.client.extended.leaderelection.LeaderElector - Successfully acquired lease, became leader
It starts leadership duties but then is quickly shutdown (since 
coordinator-1
is now up.
Copy code
2026-09-30T20:41:06,694 INFO [leader-elector-hook-worker-1] org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started lifecycle [task-master]
2026-09-30T20:55:11,449 INFO [Thread-58] org.apache.druid.indexing.overlord.DruidOverlord - I am no longer the leader...
2026-09-30T20:55:11,601 INFO [Thread-58] org.apache.druid.indexing.common.actions.SegmentAllocationQueue - Not leader anymore. Stopping queue processing.
...but there are errors:
Copy code
2026-09-30T20:55:31,627 ERROR [LeaderElectorAsyncWrapper] org.apache.druid.k8s.discovery.K8sDruidLeaderSelector - listener.stopBeingLeader() failed. Unable to stopBeingLeader: {exception
Type=java.util.concurrent.RejectedExecutionException, exceptionMessage=Task org.apache.druid.server.coordinator.loading.HttpLoadQueuePeon$$Lambda$6516/0x00007f448529a970@5641f1f0 rejecte
d from java.util.concurrent.ThreadPoolExecutor@3a4a007[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10817], class=org.apache.druid.k8s.discovery.K8s
DruidLeaderSelector}
java.util.concurrent.RejectedExecutionException: Task org.apache.druid.server.coordinator.loading.HttpLoadQueuePeon$$Lambda$6516/0x00007f448529a970@5641f1f0 rejected from java.util.concu
rrent.ThreadPoolExecutor@3a4a007[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10817]
^ this occurs right before the pod is shut down Note: I have 
terminationGracePeriod
set to 3600s (so it's gracefully shutdown and not getting a SIGTERM or anything)
Claude might have a root cause for us...
coordinator-leader-election-shutdown-race.md
Open in Slack
PreviousNext
Powered by