@Rocky what timeouts do you suggest for CN liveness and rediness?

For StarRocks Compute Nodes (CN), the suggested timeout configurations generally follow the defaults provided by the StarRocks Kubernetes Operator, with adjustments based on your specific workload and cluster size. Default Recommended Values The StarRocks Operator uses a "total failure seconds" approach to simplify probe configuration. Below are the default settings: Probe Type | Parameter | Default Value | Description Startup |

startupProbeFailureSeconds

| 300s (5 min) | Allows CN time to initialize local caches and register with FE. Liveness |

livenessProbeFailureSeconds

| 15s | Restarts the container if the CN becomes unresponsive for this duration. Readiness |

readinessProbeFailureSeconds

| 15s | Removes the CN from the service pool if health checks fail for this duration. Suggested Adjustments If you experience frequent restarts or CN nodes flipping to "NotReady" during heavy queries, consider the following optimizations: 1. Avoid Death Spirals: If your CN nodes frequently hit high CPU/memory usage, a 15s liveness timeout might be too aggressive, leading to "death spirals" where nodes restart just as they are recovering. Increasing

livenessProbeFailureSeconds

to 30s – 60s is often safer for production OLAP workloads. 2. Handle Local Cache (Shared-Data): If you use a large local disk cache, the CN may need more time to initialize the cache file system on startup. If you see the CN failing to reach a "Ready" state, increase the

startupProbeFailureSeconds

to 600s. 3. Endpoint: Use the HTTP health check endpoint rather than a TCP socket check for better accuracy: * Port:

8040

(default

be_http_port

) * Path:

/api/health

4. Graceful Exit: Ensure your

terminationGracePeriodSeconds

in Kubernetes is set higher than your StarRocks

stop_cn.sh

timeout (default ~30-60s) to allow existing queries to finish before the pod is killed. How to Configure (Operator/Helm) In your

StarRocksCluster

CRD or Helm

values.yaml

, you can set these under the `starRocksCnSpec`:

yaml
starRocksCnSpec:
  # Total seconds before the container is restarted
  livenessProbeFailureSeconds: 30 
  # Total seconds before the pod is removed from service endpoints
  readinessProbeFailureSeconds: 20
  # Total seconds allowed for the initial start
  startupProbeFailureSeconds: 300

Note: Setting these values to

0

in the Operator spec will disable the corresponding probe. References • quick_start/helm.md