We did rolling restarts yesterday and today. In sh...
# generalj
JRob
12/16/2025, 4:08 PMWe did rolling restarts yesterday and today. In short; a busy broker took 2.5 hours to restart and a non-busy broker took 4 minutes. Is there any optimization that can be made during broker start sequence to avoid needing to wait hours for startup?
We have 3 query nodes. All three are behind a LB but one of the nodes got hard-coded by a busy client. An obvious solution is to have all clients use the LB but there must be something else we can do to improve the startup time of a broker who got pinned by a client.
JRob
12/16/2025, 4:11 PMI should note that during that 2.5 hours, the client wasn't able to actually get any requests answered. So something feels a little off there...
r
Renato CRON
12/16/2025, 5:16 PMdeployed with helm ?
"mm-less" ? i think could be the middlemanagers holding until a job finishes
j
JRob
12/16/2025, 6:01 PMIt was manually restarted to apply config changes. The broker is what took forever to restart.
JRob
12/17/2025, 2:49 PMAll of the data nodes (with the middlemanagers on them) had already been restarted before tackling the broker restarts.
JRob
12/17/2025, 2:49 PMOof
Copy code
2025-12-15T21:35:02,567 INFO [main] org.eclipse.jetty.server.Server - Started @8807307msr
Renato CRON
12/17/2025, 3:20 PMas the broker is stateless i would have killed it much earlier 😆
j
JRob
12/17/2025, 3:21 PMSure, but there's no guarantee it's going to boot faster the 2nd time around 🙂
Didn't see any warnings or errors other than queries timing out.