The spike in latency percentiles (P95/P99) during a CN node restart—even when using a *graceful shutdown (

stop_cn.sh -g

)*—is a known behavior caused by several factors ranging from hardware-level caching to cluster-wide resource redistribution. While a "graceful" exit ensures that running queries aren't abruptly killed, it does not guarantee a completely transparent transition for the following reasons: 1. Cold OS Page Cache (Most Common Cause) Even if your Data Cache (disk cache) is persistent across restarts (which is the default in shared-data mode), the OS Page Cache (RAM) is completely wiped. * Impact: Reading data from RAM is orders of magnitude faster than reading from local NVMe/SSD. After a restart, the first few queries must fetch data from the disk cache (or worse, remote storage if the disk cache is cold), leading to a significant spike in response time until the OS re-populates its memory cache. 2. Immediate Rejection of New Fragments When a CN receives the

-g

signal, it transitions to a

SHUTDOWN

state. * Behavior: It stops accepting new query fragments immediately. If the Frontend (FE) hasn't fully updated its routing table yet (which can take a few heartbeats), it might still attempt to send work to that node, receive a rejection, and then have to retry on another node. This internal retry logic adds measurable latency to the end-to-end query. 3. Load Redistribution on Surviving Nodes When you take one CN node offline, its share of the workload is instantly shifted to the remaining nodes. * Impact: If your cluster is running at 60-70% utilization, losing one node (e.g., in a 3-node cluster) can push the remaining nodes toward 100% CPU/IO utilization. This "crowding" effect increases queuing time for all queries, pushing up the high-percentile latency. 4. FE Awareness & Version Differences The "graceful" nature depends heavily on the StarRocks version: * Prior to v3.4: The FE might mark a node as

DEAD

if heartbeats fail during the shutdown window, causing it to cancel queries prematurely. * v3.4 and later: The FE correctly recognizes the

SHUTDOWN

state, allowing a longer window (controlled by

loop_count_wait_fragments_finish

) for queries to finish without being cancelled. 5. RPC Reconnection Overhead When the CN restarts, all existing bRPC (internal) and client connections are severed. Re-establishing these connections—especially under high concurrency—consumes CPU cycles and adds a small "handshake" delay to the initial batch of queries post-restart. ────────── Recommendations to Mitigate Spikes: * Pre-warm the Cache: After the node is back online, run a small set of representative heavy queries to "warm up" the OS Page Cache before opening it to full production traffic. * Check

preStop

in K8s: If running on Kubernetes, ensure your

terminationGracePeriodSeconds

is longer than the StarRocks graceful timeout (default 20s). If the K8s pod is killed before the StarRocks process finishes its graceful loop, it reverts to a hard crash. * Upgrade to v3.4+: Ensure you are on v3.4.x or later to benefit from the improved FE/CN coordination during exit. * Stagger Restarts: Never restart multiple CNs simultaneously; wait for the P99 latency of the first node to stabilize before moving to the next. References • faq/Sql_faq.md • faq/Others.md • administration/Meta_recovery.md • deployment/upgrade.md • knowledge/trouble_shooting/fe_deadlock_problems.md

@Rocky But, In my case, I registered BlackList for target node before restart

Even if you manually add a CN node to the Blacklist before a restart, latency spikes can still occur due to several technical nuances in how StarRocks manages node state and query distribution. Based on how the Blacklist and Graceful Shutdown mechanisms interact, here are the most likely reasons: 1. Blacklist is Local to each FE (Crucial) In StarRocks, the BE/CN Blacklist is currently not shared across FE nodes. * The Issue: If you have multiple FEs (e.g., 1 Leader and 2 Followers) and you execute

ADD COMPUTE NODE BLACKLIST

while connected to one FE, the other FEs do not know about this blacklist. * The Result: Queries routed through other FEs will continue to send fragments to the target CN node until the process actually stops, leading to the same latency/retry issues you were trying to avoid. * Solution: You must execute the blacklist command on every FE node in the cluster. 2. Immediate Load Shift (The "Crowding" Effect) The moment a node is blacklisted, it stops accepting new query fragments. 100% of the traffic that was supposed to go to that node is instantly redirected to the remaining nodes. * The Issue: If your cluster is under high load, the remaining nodes suddenly experience a surge in CPU and memory usage. This leads to queuing at the bRPC and execution levels on the "healthy" nodes, causing a spike in P99 latency cluster-wide. * Observation: The spike doesn't happen because the restarting node is "broken," but because the remaining nodes are suddenly overworked. 3. Existing Query "Tail" Latency The Blacklist prevents new queries from starting on that node, but it does not kill existing queries. * The Issue: If you blacklist a node and then immediately issue the restart command (or if the graceful shutdown timeout is short), the node is still processing the "tail" of queries assigned before the blacklist. * The Result: If those remaining queries take longer than expected and the node eventually shuts down or slows down during its exit sequence, those specific queries will report high latency. 4. Cold Data Cache (After Restart) If the spike occurs after the node comes back online and is removed from the blacklist: * The Issue: While the CN process was down, the OS Page Cache (memory) was cleared. Even if your StarRocks Data Cache is persistent on disk, the first few queries hitting that node will be significantly slower as they read from disk instead of RAM. * Recommendation: Keep the node blacklisted for a few minutes after it starts, and run a "warm-up" script (heavy queries) to populate the cache before removing it from the blacklist. Summary Checklist: 1. Check FE consistency: Did you run the blacklist command on all FEs? 2. Monitor remaining nodes: Check if CPU/Memory usage on other CNs spikes exactly when you blacklist the target node. 3. Wait for Draining: Ensure a sufficient gap (e.g., 30-60 seconds) between blacklisting and actually triggering the restart to allow existing queries to finish. 4. Version Check: Ensure you are on v4.0 or later for official CN Blacklist support; on older versions, the behavior for CNs in the blacklist might not be as robust as it is for BEs. References • administration/management/BE_blacklist.md • sql-reference/sql-statements/cluster-management/nodes_processes/ADD_BACKEND_BLACKLIST.md • administration/management/resource_management/Blacklist.md • faq/Sql_faq.md • deployment/upgrade.md References • administration/management/BE_blacklist.md