I'm running a ZK-less, MM-less cluster in K8s and ...
# troubleshooting
j
I'm running a ZK-less, MM-less cluster in K8s and TIL that when the Overlord is restarted it shuts down every single task in my cluster. Worse, it takes all of the task metadata too (i.e. the Tasks tab in the UI is emptied). Anyone know how to fix this? The documentation states:
All jobs are natively restorable, they are decoupled from the Druid deployment, thus restarting pods or doing upgrades has no effect on tasks in flight. They will continue to run and when the overlord comes back up it will start tracking them again.
But this isn't what I'm seeing at all... 😢
✅ 1
Claude analysis of my logs:
The mechanism (directly observed, not inferred):
1. 225721 — overlord-0 (the active leader) receives its shutdown signal. Lifecycle stops the task-master stage, and three components tear down together, as a single atomic group: SupervisorManager#stop → Stopping [54] supervisors (every supervisor's Kafka monitor consumer proactively leaves its group), then TaskQueue#stop → Interrupted, exiting!, then KubernetesTaskRunner#stop → interrupts its live log-watch connections to the 3 tasks that happened to be running at that instant.
2. 225834 — overlord-1 becomes the new leader. It reports KubernetesTaskRunner - Locating [54] active tasks... Located [54] active tasks (it found 54 real, still-running K8s Jobs) but then GlobalTaskLockbox - Synced [0] locks for [0] active tasks from storage — zero rows in the metadata DB for any of them.
3. Because the new leader's supervisors see no tasks recorded as active for their datasource, every one of the 54 supervisors immediately spins up a brand-new batch of tasks from scratch — this is "Supervisors are all restarted." Whatever K8s Jobs physically survived (the 54 found) are now orphaned — untracked, duplicated by the fresh tasks, and left for the cleanup executor for jobs older than P2D to eventually reap.
Why: Druid's TaskMaster lifecycle binds SupervisorManager, TaskQueue, and KubernetesTaskRunner into one group that all stop in lockstep on any loss of leadership — there's no soft handoff where a new leader adopts the previous leader's in-flight task bookkeeping. The official "MM-less K8s" docs claim tasks survive Overlord restarts, but that's only true of the underlying K8s Jobs — the metadata-level tracking that supervisors rely on to avoid restarting is wiped every time, regardless of whether the shutdown was clean.
Important to note that I'm using the K8sTaskRunner with a PostgreSQL metadata store and have 2 Overlords & 2 Coordinators.
OMFG - we somehow missed
druid.indexer.storage.type=metadata
in 3 months of being in Prod with K8s. Somehow our migration to K8s missed this flag. A huge thorn in my side removed!
😅 1