@Rocky Context: We're running StarRocks 4.1 on GKE with the kube-starrocks operator v1.11.4. Our FE pods have terminationGracePeriodSeconds: 120 (chart default). The operator injects the following preStop hook on FE pods: /bin/bash -c ${STARROCKS_ROOT}/fe/bin/stop_fe.sh -g What we observed during a GKE maintenance node drain: 1. FE node went NodeNotReady (GKE drain initiated) 2. ~2 minutes later, pod termination started — preStop hook fired, sent SIGUSR1 3. FE correctly started returning HTTP 500 on /api/health (graceful drain confirmed, 12 probe failures over ~22 seconds) 4. FailedPreStopHook reported after ~28 seconds — well within the 120s grace period 5. Pod was SIGKILL'd → ungraceful shutdown → PD had dirty filesystem → force-detach from the old node took 2m13s before the new node could attach Question: Why does the preStop hook fail after ~28s when the grace period is 120s and normal queries finish in <200ms? Our hypothesis is that when the node is already NodeNotReady, the kubelet kills the preStop exec before it completes — is that correct? And if so, is there a way to make FE shut down cleanly in this scenario (e.g. a preStop that's more resilient to a degraded kubelet), or is the dirty-detach behaviour unavoidable when GKE force-evicts pods from a NotReady node?

Based on the symptoms you've described and the behavior of StarRocks’ graceful exit mechanism, your hypothesis is likely correct. The

NodeNotReady

state is the critical factor here, indicating that the Kubelet or the underlying node is already in a degraded state when the shutdown is attempted. 1. Why the preStop hook fails after ~28s Although your

terminationGracePeriodSeconds

is 120s, the

FailedPreStopHook

at ~28s (and subsequent SIGKILL) usually points to one of three things: * Infrastructure Timeout (GKE Node Pool Upgrade): During GKE maintenance, the Google Cloud cluster manager initiates a node drain. If the node becomes

NodeNotReady

, it often means the Kubelet has lost its connection to the API server or the OS is becoming unresponsive. If the node fails to drain within a certain internal GKE timeout (often 30–60 seconds for the node-level shutdown script), the GCE instance is terminated at the VM level. This causes an immediate SIGKILL of all containers and is why the PD remains "dirty" (BDBJE didn't finish flushing to disk). * Kubelet

exec

Timeout: In some Kubernetes versions,

exec

hooks have an internal timeout. While the grace period is 120s, the command execution itself might be capped or the Kubelet might be failing to track the process because the node is already "unhealthy." * Spot/Preemptible Instances: If you are using GKE Spot nodes, the termination notice is strictly 30 seconds. In this scenario, the 120s grace period is ignored by the infrastructure, and the node is killed at the 30s mark regardless of pod settings. 2. Is the "Dirty Detach" avoidable? When a node goes

NodeNotReady

during a drain, a "clean" shutdown is extremely difficult because the control plane can no longer coordinate with the Kubelet on that node. The force-detach (2m13s) happens because the GCE control plane waits to ensure the node is actually dead before ripping the disk away to prevent data corruption. 3. Recommendations for FE Stability on GKE To improve the resilience of FE shutdowns in these scenarios: * Enable Graceful Node Shutdown: Ensure your GKE node pools have the

GracefulNodeShutdown

feature enabled (standard in newer versions, but check the Kubelet config). This allows the Kubelet to detect the node shutdown event and allocate the grace period to pods before the VM service stops. * Verify Pod Disruption Budgets (PDB): Ensure you have a PDB for your FE pods (e.g.,

minAvailable: 2

for a 3-node FE cluster). This prevents GKE from draining multiple FE nodes simultaneously, which would lead to a cluster-wide outage even if the shutdowns are ungraceful. * Review

stop_fe.sh

Timeout: The

stop_fe.sh -g

script has a default internal timeout of 60 seconds. You can explicitly increase this in the operator's

preStop

hook if needed:

bash
    /bin/bash -c "${STARROCKS_ROOT}/fe/bin/stop_fe.sh -g --timeout 120"

Note: The script must exit before

terminationGracePeriodSeconds

expires, otherwise Kubelet will send SIGTERM/SIGKILL. * Health Check Optimization: Since your FE is returning 500 on

/api/health

, the drain is working. The issue is purely the "tail" of the process (closing BDBJE metadata). Check your

fe.log

for any "BDBJE" or "Journal" related errors during that 28s window; if the network is partitioned (

NodeNotReady

), the Leader FE might hang trying to replicate metadata one last time before exiting. Summary Your hypothesis that the

NodeNotReady

status causes the Kubelet/VM to kill the hook prematurely is likely correct. In a "NotReady" state, the node's lifecycle is no longer reliably managed by the Kubernetes grace period, but rather by the infrastructure's "force-kill" timeouts. If this is happening frequently on standard nodes, it suggests the node's OS or Kubelet is crashing before the drain finishes. If you are using Spot nodes, the 30s behavior is expected and unavoidable. References

@Tim Visser Please check which version you are using , there is a well known issue with 4.1.0, docs.starrocks.io/releasenotes/release-4.1

Yes thank you, we're running 4.0.9 because of this error we observed before indeed

One of my teams had the "latest" and after GKE recreation on weekend they found it crashing so they had to revert, better make sure they are using a pinned version..also there was an announcement of GKE that before some probe times were not respected and depending the version of GKE you are running they were going to start "enforcing them".

Yes good point! We have indeed pinned to 4.0.9 not latest, and I will look intpo the GKE probe times, good shout!

You're welcome! Here is a brief summary by AI: The Google Kubernetes Engine (GKE) release documentation details a critical enforcement update starting in GKE version 1.35 regarding how timeouts for

exec

probes (liveness, readiness, and startup probes) are handled. In older GKE versions, timeout violations for these specific probe commands were often ignored or treated as non-fatal, but they are now strictly enforced as hard probe failures. The Change: Old vs. New Behavior • GKE Version 1.34 and Earlier: If an

exec

probe command took longer than its allocated

timeoutSeconds

(which defaults to 1 second if omitted), Kubernetes reported an error or warning log event. However, the command was allowed to run to completion, and the error was not interpreted as an actual probe failure. • GKE Version 1.35 and Later: Kubernetes strictly terminates and enforces the

timeoutSeconds

limit. If the command inside the

exec

probe does not successfully finish within the timeout duration, the probe officially reports a failure. Impact on Your Pods Because older versions allowed slow or hung probe commands to slide under the radar, upgrading to 1.35 without adjusting your manifests can cause severe application disruptions: • Liveness Probes: If a liveness

exec

command times out, Kubernetes assumes the container is deadlocked and restarts it. If this happens repeatedly, your Pods will become trapped in a

CrashLoopBackOff

. • Readiness Probes: If a readiness

exec

command times out, the Pod's status changes to

Ready: False

. GKE will immediately remove the Pod from Service endpoints, cutting off all incoming network traffic. • Startup Probes: If the command times out during application boot, it counts against your failure threshold, which can cause GKE to kill and cycle the container before it ever fully starts.