@Rocky hey rocky. can you tell me what happend if cn pod sigkill with out any graceful shutdown. between cn and fe pod? is that okay in fe pvc meta or fe pod?

Hey there! I'm Rocky. Dealing with a

SIGKILL

on a Compute Node (CN) in StarRocks is generally "safe" in terms of data integrity, but it will be disruptive for your active workloads. Here is what happens between the CN and FE when a CN is terminated abruptly: 1. Impact on the Connection (CN ↔️ FE) * Heartbeat Failure: The FE periodically sends heartbeats to all CNs (usually every 5 seconds). If a CN is `SIGKILL`ed, the heartbeat will fail immediately. * Status Change: After a few failed heartbeats, the FE will mark that CN as

DEAD

or

DISCONNECTED

. You can see this status by running

SHOW COMPUTE NODES;

. * Query Failure: Any queries currently being processed by that CN will fail immediately. The user will likely see an error like

RPC error

or

Connection reset by peer

. Unlike a Graceful Shutdown, the CN doesn't get a chance to finish its "in-flight" query fragments. 2. Is it okay for FE PVC Meta or FE Pod? Yes, it is perfectly okay for the FE. * Stateless CNs: CNs are designed to be stateless. They do not manage the cluster's metadata; that is the job of the FE. * No Metadata Corruption: The FE stores the cluster metadata in its own PVC (using BDBJE). A CN crashing or being killed does not write to or affect the FE's metadata files. The only metadata change the FE performs is updating the status of that CN to "Dead," which is a standard operational procedure. * FE Pod Health: The FE pod itself will remain healthy and continue to coordinate the rest of the cluster. It will simply stop scheduling new query fragments to the killed CN. 3. What about the CN's own state? * Cache: If you are using StarRocks in a shared-data architecture, CNs often have a local disk cache. A

SIGKILL

might leave some temporary files in the cache directory, but StarRocks is designed to handle this. Upon restart, the CN will clean up or re-validate its local cache. * PVC (if any): If your CN has a PVC for logs or cache, there is no risk to the cluster's system metadata. Summary & Recommendation While a

SIGKILL

won't "break" your FE or corrupt your metadata, it’s not ideal for a production environment because: 1. Queries fail instead of finishing. 2. Load jobs might fail and require a retry. Recommendation: If you are performing maintenance or scaling, always try to use Graceful Shutdown (available from v3.3+). * In Kubernetes, ensure your

preStop

hooks are configured or use the

-g

flag with the stop scripts. * A graceful exit allows the CN to finish existing queries and tells the FE it is "exiting" so the FE can stop sending it new work before the pod actually disappears. If you have more questions about your specific version or setup, let me know! References • administration/management/graceful_exit.md • quick_start/helm.md • administration/Meta_recovery.md

@Rocky explain exactly how does it work if cn pod crushed and sigkilled, and cn pvc deleted. how fe pod detect that changed when it running somequery?

If a CN pod is crushed and SIGKILLed (non-graceful shutdown) and its PVC is deleted, here is exactly how the FE detects it and what happens to your cluster: 1. How FE detects the change during a query When a CN is `SIGKILL`ed, the process ends instantly. The FE (and other nodes) detect this through two parallel mechanisms: * Network Layer (Immediate): * For any running query, the FE has active RPC connections (usually BRPC) to the CN. When the CN process is killed, the Linux kernel on the CN node sends a

FIN

or

RST

packet to close the TCP connection. * The FE receives this network error immediately. The query will fail with an error like

RPC error: Connection reset by peer

or

Backend down

. * If the network packet is lost (e.g., the whole VM crashed), the FE will wait for the

brpc_send_plan_fragment_timeout_ms

(default 60s) before timing out the query. * Heartbeat Layer (Passive - ~15s delay): * The FE runs a

HeartbeatMgr

that pings all CNs every 5 seconds. * If a heartbeat fails, the FE retries. After 3 consecutive failures (default), the FE marks the CN as

Alive: false

. * You can see this change by running

SHOW COMPUTE NODES;

. 2. Is it okay for the FE PVC or FE Pod? Yes, it is perfectly safe. * Metadata Safety: StarRocks metadata is managed solely by the FE and stored in the FE's own PVC (using BDBJE). CNs are stateless and never write to the cluster's metadata. * No Corruption: A CN crashing cannot "corrupt" the FE. The only thing the FE does is update its internal state to reflect that the CN is no longer available. This is a standard in-memory update that then gets persisted to the FE logs as a "Node Down" event. 3. What happens when the CN PVC is deleted? In a standard StarRocks Kubernetes deployment (like using the StarRocks Operator): * CN Statelessness: The CN PVC is typically used for Logs and Local Data Cache (Data Cache is used in shared-data mode to speed up object storage access). * PVC Deletion Impact: * Logs: You lose the historical logs of that specific pod. * Data Cache: You lose the local cached data from S3/GCS/Azure. When a new CN pod starts, it will have an empty cache and will need to pull data from the object storage again. This will cause a temporary "cold start" performance dip for queries, but no data is lost because the source of truth is the remote object storage. * Registration: When the CN pod restarts (even with a new PVC/IP), it will re-register or the FE will re-establish a heartbeat based on the service discovery. Summary of the "Crush" Sequence: 1. CN Pod killed: Process dies, TCP connections drop. 2. Running Queries: FE receives

RPC Error

→ Client gets

Query Failed

. 3. FE Heartbeat: Fails 3 times (~15s) → CN marked

Dead

. 4. FE Global State: FE stops scheduling new fragments to that CN IP. 5. CN PVC Deleted: Local cache/logs gone. 6. K8s Recovery: K8s recreates the CN pod. 7. Re-Join: New CN starts → FE detects heartbeat success → CN marked

Alive

. Queries start running on it again (with a cold cache). Key takeaway: The system is designed to be resilient to this. The FE remains stable, and your data (in object storage) remains safe. Only "in-flight" queries at the exact moment of the crash are lost. References • administration/management/BE_blacklist.md • administration/management/graceful_exit.md • administration/management/FE_configuration.md • deployment/deploy_manually.md