JRob
10/23/2026, 12:41 PMAll jobs are natively restorable, they are decoupled from the Druid deployment, thus restarting pods or doing upgrades has no effect on tasks in flight. They will continue to run and when the overlord comes back up it will start tracking them again.But this isn't what I'm seeing at all... 😢
JRob
10/23/2026, 12:55 PMThe mechanism (directly observed, not inferred):
1. 225721 — overlord-0 (the active leader) receives its shutdown signal. Lifecycle stops the task-master stage, and three components tear down together, as a single atomic group: SupervisorManager#stop → Stopping [54] supervisors (every supervisor's Kafka monitor consumer proactively leaves its group), then TaskQueue#stop → Interrupted, exiting!, then KubernetesTaskRunner#stop → interrupts its live log-watch connections to the 3 tasks that happened to be running at that instant.
2. 225834 — overlord-1 becomes the new leader. It reports KubernetesTaskRunner - Locating [54] active tasks... Located [54] active tasks (it found 54 real, still-running K8s Jobs) but then GlobalTaskLockbox - Synced [0] locks for [0] active tasks from storage — zero rows in the metadata DB for any of them.
3. Because the new leader's supervisors see no tasks recorded as active for their datasource, every one of the 54 supervisors immediately spins up a brand-new batch of tasks from scratch — this is "Supervisors are all restarted." Whatever K8s Jobs physically survived (the 54 found) are now orphaned — untracked, duplicated by the fresh tasks, and left for the cleanup executor for jobs older than P2D to eventually reap.
Why: Druid's TaskMaster lifecycle binds SupervisorManager, TaskQueue, and KubernetesTaskRunner into one group that all stop in lockstep on any loss of leadership — there's no soft handoff where a new leader adopts the previous leader's in-flight task bookkeeping. The official "MM-less K8s" docs claim tasks survive Overlord restarts, but that's only true of the underlying K8s Jobs — the metadata-level tracking that supervisors rely on to avoid restarting is wiped every time, regardless of whether the shutdown was clean.
JRob
10/23/2026, 1:08 PMJRob
10/23/2026, 3:08 PMdruid.indexer.storage.type=metadata in 3 months of being in Prod with K8s.
Somehow our migration to K8s missed this flag.
A huge thorn in my side removed!