[ StarRocks CN Restart - Delay Propagation / v4.0.1 ] Hey folks, I have a question regarding CN scale-in or Restart behavior. While operating CN nodes, I occasionally perform scale-in. During this process, I consistently observe a spike where P75 latency increases to tens of seconds for a short period, and at the same time, CPU usage on one of the remaining CN nodes rises sharply. For scale-in, I’m using

stop_cn.sh -g

to ensure a graceful shutdown of running queries. I’ve also enabled

graceful_exit_wait_for_frontend_heartbeat = true

, so that the CN node propagates its SHUTDOWN status to the FE and stops receiving new queries before it goes down. Based on this setup, I expected only a brief impact (0–5 seconds) due to the FE heartbeat interval. However, in practice, the latency impact seems to last up to ~2 minutes. Has anyone experienced something similar? Or are there specific areas (e.g., caching, load redistribution) that I should take a closer look at? Thanks in advance 🙏

mostly cause by caching, when doing a scale-in, the tablets on the node will be migrated to other nodes which cold data will be read. graceful shutdown can just promise the query not killed due the ongoing offline node, but can't promise the cold data /cache miss due to tablet migration.

One thing that seems odd is that when we scale out, the latency doesn’t spike much—it stays within a few hundred milliseconds. If tablets are redistributed during scale-out as well, shouldn’t we observe similar latency spikes in that case too? 🤔

yes, should be similar. scale-in can be considered as urgent tablet migration, the node will be gone in less than a minute while the scale-out can be considered as planned migration, the tablet balance can be lasted from minutes to hours.

When the status becomes SHUTDOWN, is it automatically added to the blacklist?

yes

when a CN goes down, does the FE select a replacement CN using something like a consistent hashing ring? If so, is it possible that all queries that were previously handled by the shutdown CN get concentrated on a specific “neighbor” node? I originally assumed that the load would be redistributed evenly, rather than being funneled into a single backup CN.

yes, before the tablets are all balanced to new locations, FE will select a consistent backup node the replace the offline node if no live nodes in hand.

Ah, I see..! In that case, a specific node would experience S3 I/O, and some queries could be delayed. Once the redistribution is complete, will the load be automatically spread across multiple nodes rather than being concentrated on a single node?

can enable query profile and do further investigation when query latency spikes during scale-in time window.

Ah, I’m a bit concerned that enabling query profiles in a production environment might put additional load on the FE nodes. Does it consume a lot of resources?

not that much, maybe just big queries, and set a proper big query threshold.

Great! Thanks kevin 😁

.