@Rocky deployed in AKS with helm my CN liveness and readiness probes fail often, I changed the probefailureseconds to 120 to stop causing CN restarts, is expected for these probes to fail so often?

It is not uncommon for StarRocks Compute Node (CN) liveness and readiness probes to fail under heavy load or resource constraints in a Kubernetes environment like AKS. In fact, the StarRocks Operator explicitly added configuration fields to handle this scenario. 1. Why Probes Often Fail in AKS/Kubernetes Frequent probe failures for CN nodes are typically caused by one of the following: * Heavy Workload & CPU Throttling: Under high query or load pressure, the CN process may hit its CPU limits. Because Kubernetes enforces CPU limits using cgroups, the process can be throttled, causing even a simple health check response to take longer than the default timeout (often 5-15 seconds). * Default Thresholds are Tight: The default

livenessProbeFailureSeconds

and

readinessProbeFailureSeconds

in the StarRocks Helm chart are often set to 15 seconds. This is frequently too short for high-performance OLAP nodes under stress. * Disk Cache IO Pressure: CN nodes use disk space for caching. If the node is struggling with IO or if multiple CNs are scheduled on the same AKS node without enough throughput, the heartbeat/health check might lag. 2. Validating Your Solution Increasing the failure seconds is the recommended approach in official StarRocks documentation and release notes (specifically since Operator v1.8.7). If you are using the StarRocks Operator/Helm chart, you should ensure you are setting these specific fields in your

values.yaml

or

StarRocksCluster

CRD:

yaml
starRocksCnSpec:
  livenessProbeFailureSeconds: 120
  readinessProbeFailureSeconds: 120

3. Recommendations While 120 seconds helps stop the "restart loop," you may want to investigate the following to address the root cause: * Check CPU Throttling: Run

kubectl top pods

or check Azure Monitor/Prometheus metrics to see if your CN pods are consistently hitting their CPU

limits

. If they are, consider increasing the CPU requests/limits or using higher-performance AKS node SKUs. * Startup Probes: Ensure you have a

startupProbe

configured. If the failures happen primarily during node startup or scale-up, a startup probe with a high

failureThreshold

(e.g., 300s) prevents the liveness probe from killing the pod before it finishes initializing its cache and metadata. * Pod Anti-Affinity: If you have multiple CNs on one AKS node, they may compete for the same disk/network resources. Use

podAntiAffinity

to ensure CN pods are spread across different physical nodes. Summary: It is expected behavior under stress. Your change to 120 seconds is a standard practice to improve stability in production Kubernetes environments. References • deployment/deploy_manually.md • quick_start/helm.md • deployment/sr_operator.md