http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> All 8 BE nodes hit 100% memory on a...
# questions-and-troubleshooting
n
@Rocky All 8 BE nodes hit 100% memory on a query that was pruned to a single bucket and cluster timing out all queries. Setup Shared-nothing cluster steup: StarRocks v4.0.10, 8 BE nodes and 3 FE nodes. Hosted on aws eks. What happened: A dev accidentally fired a query containing a pathological LIKE pattern — a literal prefix followed by thousands of consecutive % (e.g. LIKE '%ashley%si%%%%…(2,000+ %)…%b%'). The same pattern is also used inside any_match(array_map(x -> x LIKE '…', emails)) on an ARRAY<VARCHAR> column, plus on two scalar columns, combined with OR. We ran it a couple of times. Each run effectively timed out, but cpu and memory climbed steadily on all 8 BEs up to ~100% — not a clean Memory limit exceeded cancel, just RSS rising until the nodes became unresponsive. 4+ queries hit 90s timeout, saturated BE worker pool → BE thrift response queue stalled → FE socket write. After 4×90s queries jammed BE worker pool, dependent queries timed out at 60s, Same agent_contact_base join family. Some contact_conversation JOIN outreach_intelligence_contact_entry (no segment pruning) scanned 4.4M rows / 168 MB repeatedly. This caused all queries failing to starrocks and we have to manually restart all BE worker nodes/pods. We are using default pod liveness/readiness http probes on `path: /api/health' and 
port: 8040
and during whole time the cluster was timing out all queries, its probes were passing so nodes/pods were up, now we want to understand what should we do add/update to config so such scenerio dont happen and it should automaticaly handle such situtaions and not to become unresponsive only due to bad queries timing out and makw the whole cluster unresponsive. Also want to understand together with config, should we add some custom health-check probes to BE/FE nodes/pods and what should we check/implement in that probes?
r
The pathological 
LIKE
pattern (2,000+ wildcards) likely triggered a worst-case complexity scenario in the expression evaluation engine, causing extreme CPU consumption and memory growth that bypassed standard query limits. Because the 
/api/health
endpoint on port 8040 is handled by a separate web server thread, it remained responsive even while the execution worker pool was completely saturated. To prevent this from taking down your cluster in the future, you should implement Resource Isolation, Query Queues, and Deep Health Checks. 1. Implement Resource Group Circuit Breakers Resource groups can automatically identify and "kill" runaway queries based on CPU time or memory usage before they impact the entire node. Create a resource group (or update your default) with strict "Big Query" limits:
Copy code
sql
ALTER RESOURCE GROUP default_wg WITH (
    'big_query_cpu_second_limit' = '300', -- Kill any query using > 5 mins of CPU on a single BE
    'big_query_mem_limit' = '10737418240' -- Kill any query using > 10GB RAM on a single BE
);
* Why this works: Unlike session variables which rely on the query "cooperating," these are enforced by the Pipeline Engine's scheduler. If a query exceeds these, the BE will forcibly terminate the fragment. 2. Enable Query Queuing When a BE is under high stress, it should stop accepting new work. Query queues prevent the "Thrift worker pool saturation" you experienced by queuing or rejecting queries once a resource threshold is hit. Enable these global variables:
Copy code
sql
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL query_queue_cpu_used_permille_limit = 800; -- Queue if CPU > 80%
SET GLOBAL query_queue_mem_used_pct_limit = 0.85;    -- Queue if RAM > 85%
SET GLOBAL query_queue_max_queued_queries = 100;    -- Reject after 100 in queue
* Why this works: It prevents new queries from entering the worker pool and stalling the Thrift response queue when the BE is already struggling. 3. Implement "Deep" Liveness Probes The default 
/api/health
only checks if the HTTP server is up. For EKS, you should augment this with a probe that verifies the execution engine is alive. Recommendation: Script-based Exec Probe Add a sidecar or a custom 
livenessProbe
in your Kubernetes manifest that attempts a trivial SQL query through the FE or checks BE internal metrics. Since you are on BE nodes, you can probe the internal metrics for worker pool saturation:
Copy code
yaml
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      # Check if the number of running queries is at a "deadlock" level 
      # or if the BE is responding to local metric requests within 5s
      curl -sSf --max-time 5 <http://localhost:8040/metrics> | grep starrocks_be_query_scan_bytes_total
  initialDelaySeconds: 60
  periodSeconds: 30
Alternatively, use a Readiness Probe that queries the FE: 
SELECT 1
. If the cluster is jammed, the FE will timeout, the pod will be marked "Not Ready," and the Load Balancer will stop sending traffic to that node. 4. Adjust BE Worker Pool (be.conf) If your BE worker pool (
be_service_threads
) is too large, it can lead to excessive context switching during high load. If it's too small, it jams easily. * Check your 
be_service_threads
(Default: 64). In v4.0+, ensure this is balanced against your CPU cores. * If you encounter "Thrift response queue stalled," consider decreasing 
thrift_server_max_worker_threads
(in 
fe.conf
) or increasing 
be_service_threads
to handle more concurrent RPCs, but only if you have implemented the Resource Group limits above first. Summary of Config Changes Component | Setting | Recommended Action Resource Group | 
big_query_cpu_second_limit
| Set to a reasonable ceiling (e.g., 300-600s). Resource Group | 
big_query_mem_limit
| Set to ~50% of BE RAM to prevent a single query from OOMing a node.
Global Variable | 
query_queue_cpu_used_permille_limit
| Set to 800 (80%) to stop "piling on" load. Global Variable | 
query_mem_limit
| Ensure this is set (e.g., 10G) so 
LIKE
operations don't grow RSS indefinitely. K8s Probe | 
livenessProbe
| Move from 
tcpSocket
to a 
curl
against 
/metrics
with a short timeout. References • administration/management/resource_management/query_queues.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/BE_configuration.md
n
@Rocky can you explain how does this helps with liveness?
Copy code
yaml
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      # Check if the number of running queries is at a "deadlock" level 
      # or if the BE is responding to local metric requests within 5s
      curl -sSf --max-time 5 <http://localhost:8040/metrics> | grep starrocks_be_query_scan_bytes_total
  initialDelaySeconds: 60
  periodSeconds: 30
r
This specific 
livenessProbe
helps by shifting the check from a "Is the process running?" check to a "Is the process capable of performing work within a reasonable timeframe?" check. Here is how it addresses the "unresponsive but passing health checks" scenario you encountered: 1. Enforcement of Latency (The 
--max-time 5
factor) The default 
/api/health
check is extremely "shallow." It often just returns a static string from the web server thread. Even if the BE is at 100% CPU and its execution threads are deadlocked, the web thread might still respond to a simple ping in milliseconds. By using 
curl --max-time 5
, you are telling Kubernetes: "If the BE is so saturated that it cannot respond to a local request within 5 seconds, consider it dead." In your scenario, where the BE worker pool was jammed and RSS was hitting 100%, internal resource contention would likely cause the metrics endpoint to hang or slow down significantly, triggering this timeout. 2. "Deep" vs "Shallow" Health * Shallow (
/api/health
): Simply checks if the HTTP server is listening. It rarely fails unless the process actually crashes. * Deep (
/metrics
): To serve the 
/metrics
endpoint, the BE must collect statistics from various internal modules (Buffer pool, Fragment Manager, Scan nodes). If the BE is experiencing internal "gray failure" (where threads are stuck waiting for memory or CPU), the process of gathering these metrics will stall. * The `grep`: Adding 
grep starrocks_be_query_scan_bytes_total
ensures that the BE didn't just return an empty response or an error message; it proves the internal monitoring system is still functional. 3. Automated Recovery from "Zombie" States In your case, you had to manually restart the BE pods. * With this 
livenessProbe
, Kubernetes performs that manual restart for you. * When the pathological query pins the CPU/Memory and the probe fails 3 times (the default 
failureThreshold
), Kubernetes sends a 
SIGTERM
(and eventually a 
SIGKILL
) to the pod. * This clears the jammed memory and worker pool automatically, restoring cluster health without human intervention. 4. Why use 
exec
instead of 
httpGet
? While Kubernetes has a built-in 
httpGet
probe, using 
exec
with 
curl
inside the container is often more reliable for StarRocks on EKS because: 1. Network Independence: It rules out issues with the K8s service mesh or CNI; the check is performed over the loopback interface inside the pod. 2. Tooling: It allows you to use the 
--max-time
flag explicitly, which is the most critical setting for detecting a "frozen" node that hasn't actually crashed. Recommendation for your Setup Since you are on v4.0.10, you can make this even more effective by checking a specific metric that indicates worker saturation. If 
starrocks_be_short_key_index_mem_bytes
or other core metrics can't be retrieved, the node is definitely unhealthy. Caution: Ensure 
initialDelaySeconds
is long enough (e.g., 60-120s) so that the BE doesn't get caught in a "boot loop" because it's busy loading metadata or tablet headers during startup.
p
@Rocky Does FE use any health checking mechansim on BE before issuing tasks?
r
Yes, the FE uses several levels of health checking and status monitoring before issuing tasks to a BE. However, by default, these checks are primarily "liveness" checks (is the process up?) rather than "load-aware" checks. Here is the breakdown of the mechanisms StarRocks uses: 1. Heartbeat Mechanism (Liveness) The FE sends a heartbeat to every BE every 5 seconds. * Threshold: If a BE fails to respond to 3 consecutive heartbeats, the FE marks the BE as 
DEAD
. * Behavior: Once a BE is marked 
DEAD
, the FE will stop scheduling any new queries to it and will eventually begin re-replicating tablets stored on that node to other healthy nodes. * The Issue: A heartbeat is a very "shallow" RPC. Even if the BE's worker pool is 100% saturated with a pathological query, the thread handling the heartbeat (usually a dedicated heartbeat service thread) might still respond, leading the FE to believe the BE is healthy. 2. Resource Usage Reporting (Performance Monitoring) By default, every BE reports its current resource status to the Master FE every 1 second (controlled by 
report_resource_usage_interval_ms
in 
be.conf
). This report includes: * CPU usage (permille) * Memory usage (bytes) * Number of currently running queries Crucially, the FE does not use this information to skip a BE unless you have enabled Query Queues. If Query Queues are disabled (the default), the FE will continue to send tasks to a BE even if it reports 99% CPU and 99% memory usage. 3. Query Queues (Load-Aware Scheduling) This is the mechanism that would have prevented your cluster-wide timeout. When enabled, the FE checks the most recent resource report from the BEs before issuing a new query. If you set:
Copy code
sql
SET GLOBAL query_queue_cpu_used_permille_limit = 800; -- 80% CPU
SET GLOBAL query_queue_mem_used_pct_limit = 0.8;    -- 80% RAM
The FE will check the BE's reported load. If the BE is already saturated, the FE will either queue the query or return an error immediately rather than sending more work to a "dying" BE and clogging the Thrift worker pool. 4. BE/CN Blacklist (Fault Tolerance) The FE maintains an internal "Blacklist" for nodes that are technically "alive" but failing to process work. * Trigger: If a query fragment sent to a BE fails due to a connection timeout or a network error, the FE increments a failure count for that node. * Action: If failures exceed a threshold within a certain window (
black_host_connect_failures_within_time
), the FE will temporarily stop sending new query fragments to that specific BE. * Recovery: The FE continues to probe the blacklisted node and will only re-enable it if a series of probes succeed. Why it failed in your case: In your scenario with the pathological 
LIKE
pattern: 1. Heartbeat succeeded: The BE process was technically "up," so heartbeats were likely still being acknowledged. 2. No Queueing: Since 
query_queue_..._limit
variables were likely at their default (
0
), the FE ignored the fact that BEs were at 100% load and kept "piling on" new queries. 3. Worker Pool Saturation: The BE's 
service_threads
(Thrift pool) became completely occupied waiting for the pathological queries to process. When the FE tried to send new queries, the BE couldn't even "accept" the RPC, leading to the socket write timeouts you saw. Recommendation To make the FE "smarter" about your BE health, you must enable the Resource Thresholds for Query Queues. This forces the FE to treat a saturated BE as "busy" rather than "healthy," preventing the death-spiral you experienced by your worker pool saturation of the whole cluster when one or more nodes are struggling. References • administration/management/resource_management/query_queues.md
p
@Rocky Does FE honour the BE pod kubelet reported ready status or it will rely purely on its own mechanism?
r
The short answer is no. The StarRocks Frontend (FE) does not honor the Kubelet's "Ready" status for query scheduling. It relies purely on its own internal heartbeat and resource reporting mechanism. In your scenario, this is precisely why the cluster became unresponsive even though the pods were "Running" and "Ready" in Kubernetes. How the FE and Kubelet Differ There are two independent health-check loops running in your EKS cluster: Feature | ***Kubernetes Kubelet (Readiness)*** | ***StarRocks FE (Heartbeat)*** Target | The Pod's container status. | The StarRocks BE process. Check Type | HTTP GET on 
/api/health
(Shallow). | RPC Heartbeat on 
heartbeat_service_port
(Shallow). Action on Failure | Removes Pod from the K8s Service (ClusterIP/LB). | Marks BE as 
DEAD
in 
SHOW BACKENDS
. Query Scheduling | Irrelevant. FE talks to BEs directly via IP/FQDN. | Critical. FE only schedules tasks to "Alive" nodes. Why this caused your "Death Spiral" 1. Direct Communication: When an FE schedules a query, it does not go through a Kubernetes Service to reach the BE; it sends the task directly to the BE's internal IP. Therefore, even if the Kubelet marks a pod as 
Not Ready
, the FE will continue to send it query fragments as long as the internal StarRocks heartbeat passes. 2. Shallow Heartbeats: The BE heartbeat is handled by a dedicated lightweight thread. Even when the BE worker pool is 100% jammed by a pathological 
LIKE
query, that heartbeat thread often remains responsive enough to satisfy the FE. 3. The Result: The FE thinks the BE is "Alive," the Kubelet thinks the Pod is "Ready" (because the HTTP server is up), but the BE's actual query execution threads are effectively deadlocked. How to fix the "K8s-Blindness" To ensure the FE stops sending work to a pod that Kubernetes (or the node itself) knows is struggling, you must bridge this gap using the following configurations: 1. Enable Load-Aware Scheduling (FE Side) Enable Query Queues. This is the only way to make the FE "honor" the actual health of the BE. When enabled, the FE will look at the BE's reported CPU/Memory (sent via an internal report every 1s) and stop sending new fragments if thresholds are exceeded.
Copy code
sql
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL query_queue_cpu_used_permille_limit = 800; -- 80% CPU
SET GLOBAL query_queue_mem_used_pct_limit = 0.85;    -- 85% RAM
2. Deepen the K8s Liveness Probe (Kubelet Side) Since the FE won't check the Kubelet, you must make the Kubelet more aggressive so it restarts the pod the moment it becomes unresponsive. As discussed, move from a shallow HTTP check to a "Deep" check that verifies the BE can still perform internal tasks:
Copy code
yaml
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      # If the BE can't even respond to a local metrics request in 5s,
      # it's better to kill it and let it restart than keep it in the cluster.
      curl -sSf --max-time 5 <http://localhost:8040/metrics> | grep starrocks_be_query_scan_bytes_total
3. Shorten the Heartbeat Timeout (BE/FE Side) You can make the internal StarRocks heartbeat more sensitive so the FE marks the BE as 
DEAD
faster: * FE Config (
fe.conf
): 
heartbeat_timeout_second
(Default: 30s). Reducing this to 10-15s will help the FE realize a node is "gone" much sooner. * BE Config (
be.conf
): 
heartbeat_service_thread_count
(Default: 1). If you have many FEs, ensure the BE has enough threads to answer heartbeats even under load. Summary The FE is "K8s-blind." To prevent this from happening again, don't rely on Kubernetes to tell StarRocks the node is healthy; instead, enable Query Queues so the FE uses its own internal load metrics to protect the cluster from runaway queries that saturate the worker pools.
p
@Rocky Weird, is starrocks really a k8s native implementation? Can you share the reference for the below statement in your previous answer?
The short answer is no. The StarRocks Frontend (FE) does not honor the Kubelet's "Ready" status for query scheduling. It relies purely on its own internal heartbeat and resource reporting mechanism.
r
The term "K8s native" can be nuanced. StarRocks is K8s-native in terms of orchestration (via the StarRocks Operator), but its query execution engine maintains its own internal health-check loop for performance and architectural reasons. The FE's "blindness" to the Kubelet status is a design choice common in high-performance distributed databases. Here are the references and the architectural reasoning: 1. The Internal "Source of Truth" (Reference: FAQ & Architecture) According to the StarRocks Operation and Maintenance FAQ:
"When Heartbeat (default: every 5 seconds) failed 3 times, the BE is marked as not alive."
This internal heartbeat is the only signal the FE's 
HeartbeatMgr
uses to update the 
Alive
status in 
SHOW BACKENDS
. The FE does not query the Kubernetes API Server to check Pod Readiness before scheduling a query because: * Latency: Query scheduling happens in milliseconds. Querying the K8s API for the status of 100+ pods for every SQL query would introduce prohibitive latency and put immense load on the K8s Control Plane. * Direct Routing: The FE routes query fragments directly to BE IPs (or FQDNs). It bypasses Kubernetes Services (ClusterIP) for internal data transfer to avoid the overhead of 
kube-proxy
or the CNI's load-balancing layer. 2. Resource Reporting Mechanism (Reference: BE Configuration) The BE reports its own health and load directly to the FE using the `report_resource_usage_interval_ms` (default: 1000ms). * This report includes CPU usage, memory usage, and the number of running queries. * The FE stores this in memory. If you enable Query Queues, the FE will finally "honour" this data and stop sending queries to a BE that reports 90% CPU usage, even if the pod is "Ready" in Kubernetes. 3. The Role of the StarRocks Operator The StarRocks Operator is responsible for the Lifecycle, not the Execution: * Lifecycle: If a Pod is deleted or a Node fails, the Operator/StatefulSet handles the restart. When a new Pod comes up, the Operator (or init script) runs 
ALTER SYSTEM ADD BACKEND
to register it with the FE. * Execution: Once registered, the FE takes over. It assumes that if it can reach the BE via RPC and the heartbeat responds, the BE is capable of working. Why this doesn't make it "non-native" StarRocks is designed to run in environments where the infrastructure (like K8s) might be slower to react than the database needs to be. * Kubelet Readiness is designed for North-South traffic (e.g., an Ingress controller deciding whether to send a web request to a pod). * StarRocks Heartbeats are designed for East-West traffic (distributed execution fragments). Summary Reference: You can verify this behavior by running 
SHOW BACKENDS;
while a pod is in a 
Running
but 
Not Ready
state (e.g., during a slow startup). You will often see 
Alive: true
in StarRocks before the Kubelet marks the pod as 
Ready
, or vice versa during a heavy-load "gray failure." To bridge this, the 
livenessProbe
you found is the standard "K8s-native" way to force the infrastructure to intervene when the application-layer health checks (heartbeats) are too shallowly pass but the application is actually stuck. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md
n
@Rocky We are running StarRocks 4.0.10 on Kubernetes/EKS in shared-nothing mode. Our BE pods currently have HTTP probes on 
/api/health
port 8040 for liveness, readiness, and startup. During an incident, several catastrophic LIKE-pattern queries saturated BE worker pools, FE saw Broken pipe errors, and responsive BEs were blacklisted. The BE HTTP health endpoint may still be reachable in some overload cases, so I do not think a liveness probe that only checks 
/metrics
or greps for 
starrocks_be_query_scan_bytes_total
is enough. Can you validate the best Kubernetes health-check strategy for this case? Specifically: 1. Is a liveness probe like this actually meaningful, or does it only prove the BE HTTP metrics endpoint is alive?
Copy code
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      curl -sSf --max-time 5 <http://localhost:8040/metrics> | grep starrocks_be_query_scan_bytes_total
  initialDelaySeconds: 60
  periodSeconds: 30
2. Should liveness remain a conservative local-only check, for example 
/api/health
or 
/metrics
, with high failureThreshold, so Kubernetes only restarts the BE if the local BE process/HTTP server is truly stuck? 3. Should deeper checks be moved to readiness instead of liveness? For example, readiness could check: local BE HTTP health succeeds, FE can see this BE in 
SHOW BACKENDS
, this BE has 
Alive = true
, this BE is not present in 
SHOW BACKEND BLACKLIST
, optionally reject readiness if NumRunningQueries / CPU / memory are above a very high emergency threshold. 4. Is it safe/recommended to run 
SELECT 1
through FE from a BE readiness probe, or should SQL-through-FE checks be done by an external watchdog/CronJob instead of per-BE liveness? My concern is that FE/network slowness could cause Kubernetes to restart healthy BE pods. 5. Which exact StarRocks 4.0.10 metrics from 
<http://BE:8040/metrics>
are best for detecting a BE execution-engine stall or worker-pool saturation? I am looking for actual metric names and safe thresholds. For example: running queries, query execution threads / worker pool queue length, fragment execution queue, RPC/thrift queue saturation, query timeout/error count, BE CPU/memory pressure. 6. For this incident type, would you recommend this split? liveness: local 
/api/health
or 
/metrics
, conservative, restart only after 1–2 minutes of failure; readiness: local health + 
SHOW BACKENDS
+ `SHOW BACKEND BLACKLIST`; external watchdog: periodically checks 
SHOW RUNNING QUERIES
, 
SHOW PROC '/current_queries'
, 
SHOW BACKENDS
, query queue metrics, and kills known-bad/stuck queries before restarting pods; StarRocks query queue/resource group settings for prevention. explain which checks should be liveness vs readiness vs external watchdog.
r
In StarRocks 4.0.10, relying on Kubelet status is inherently insufficient because the BE's internal query scheduling is decoupled from Kubernetes readiness. Here is the validation of your strategy and the recommended metrics for detecting execution engine stalls. 1. Meaning of Liveness Probe The 
curl ... /metrics | grep ...
probe is marginally better than 
/api/health
because it proves the metrics subsystem is active, but it does not prove the execution engine is healthy. In a saturated scenario, the BE HTTP server (handled by a dedicated web server thread) can still serve metrics while all query execution threads are deadlocked or 100% occupied by a complex 
LIKE
pattern. * Recommendation: Use it only as a "process is alive" signal. Do not expect it to catch query-level hangs. 2. Liveness: Conservative Local-Only Check Yes. Liveness probes should remain conservative (e.g., 
/api/health
). * Reason: StarRocks is a stateful system. A K8s restart triggers a complex series of internal events: FE detects the loss, starts tablet migration (re-balancing), and then must re-sync metadata when the BE returns. * Threshold: Use a high 
failureThreshold
(e.g., 5–10 mins). Only restart the BE if the process is completely unresponsive. For query saturation, killing the query is better than killing the pod. 3. Readiness: Deeper Checks No, do not move FE-dependent checks (like 
SHOW BACKENDS
) into the BE readiness probe. * The Circular Dependency Danger: If the FE is slow or the leader is switching, all BE readiness probes will fail simultaneously. This causes Kubernetes to remove all BEs from service endpoints, which could break external load balancers or metric scrapers, compounding the outage. * Readiness in StarRocks: Since the FE schedules queries directly to BE IPs (bypassing K8s Services), a BE failing its readiness probe does not stop the FE from sending it more queries. Readiness probes on BEs only affect traffic coming through the K8s service (e.g., external monitoring or specific proxies). 4. SELECT 1 from BE Probe Not recommended. Running SQL through the FE from a per-BE probe is a significant risk. If you have 50 BEs all running a 
SELECT 1
every 10 seconds, and the FE becomes slightly slow, you risk a "Readiness Storm" where the entire cluster goes 
Unready
exactly when you need it to be stable. 5. Critical StarRocks 4.0.10 Metrics (BE Execution Engine) Since StarRocks 4.0 uses the Pipeline Engine by default, monitor these metrics from 
<http://BE:8040/metrics>
to detect saturation: Metric Name | Type | Critical Threshold / Meaning 
starrocks_be_pipe_driver_queue_len
| Gauge | Most Important. If this is high (>100), the BE has "Ready" tasks that cannot get a CPU core. 
starrocks_be_pipe_poller_block_queue_len
| Gauge | High values indicate tasks are blocked (likely on I/O or network). 
starrocks_be_exec_state_report_queue_count
| Gauge | Threshold: 1000. If this hits 1000, the BE cannot report its status to the FE, leading to "Broken Pipe" or blacklisting. 
starrocks_be_cpu_used_permille
| Gauge | Threshold: > 900. 900 means 90% CPU usage across all cores. 
starrocks_be_num_running_queries
| Gauge | Compare against your 
query_queue_concurrency_limit
. 6. Recommended Health-Check Split Component | Responsibility | Recommended Config / Logic Liveness Probe | Process survival | Local 
/api/health
. Conservative. Threshold: 120s+ failure. Readiness Probe | Basic data path | Local 
/api/health
. Aggressive (shorter timeout) to pull from LB if needed. Query Queues | Prevention | Crucial. Set 
query_queue_cpu_used_permille_limit = 850
. This stops FE from sending work to saturated BEs. Resource Groups | Protection | Create a "Big Query" group for 
LIKE
patterns with 
big_query_cpu_second_limit
.
Watchdog | Mitigation | A script running against the FE that monitors 
SHOW PROC '/current_queries'
and kills queries running > X seconds with high CPU usage. Final Recommendation: Instead of trying to make Kubernetes "smarter" via probes, enable Query Queues on the FE. This is the native StarRocks mechanism that ensures the FE "honours" the health and CPU load of the BE before issuing a task. References • administration/management/monitoring/metrics.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md References • administration/management/monitoring/metrics.md • administration/management/resource_management/query_queues.md
n
@Rocky so can you give me exact livenessprobe that i can go with, which can take care of my issues and suits best for us
r
For StarRocks 4.0.10, the most robust strategy is to move away from a "shallow" HTTP check to a "Semi-Deep" Scripted Probe. This probe verifies that the BE process is running AND that the internal communication channel back to the FE isn't completely blocked. The "Execution-Aware" Liveness Probe This script doesn't just check if the web server is up; it checks if the report queue (the BE's mailbox to the FE) is full. When this queue hits 1000, the BE is effectively "brain-dead" to the rest of the cluster because it can no longer report fragment status, leading to the "Broken pipe" and blacklisting you saw.
Copy code
yaml
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      # 1. Check basic process health
      curl -sSf <http://localhost:8040/api/health> || exit 1
      
      # 2. Check the FE Report Queue. 
      # If this is stuck at 1000, the BE is "ghosting" the FE and should be restarted.
      REPORT_QUEUE=$(curl -s <http://localhost:8040/metrics> | grep 'starrocks_be_exec_state_report_queue_count' | awk '{print $2}')
      if [ "$REPORT_QUEUE" -ge 1000 ]; then
        echo "BE Report Queue is full ($REPORT_QUEUE). BE is unresponsive to FE."
        exit 1
      fi
  initialDelaySeconds: 60
  periodSeconds: 30
  failureThreshold: 4  # Total 2-minute "grace" period before K8s kills the pod
  timeoutSeconds: 5
Why this is the "Best" Strategy for your Incident: 1. Detection of the "Death Spiral": During your incident, the BE worker pools were saturated. The metric 
starrocks_be_exec_state_report_queue_count
specifically tracks the backlog of status reports. If this is full, the FE cannot know if a query fragment finished, causing the "Broken pipe" errors you experienced. 2. Avoids False Restarts: By using a 
failureThreshold
of 4 and a 
periodSeconds
of 30, you give the BE 2 minutes to recover from a transient spike. A restart is expensive (metadata re-sync, tablet balancing), so we only pull the trigger if the blockage is persistent. 3. Conservative Readiness: Keep your Readiness Probe simple (
/api/health
). * Reason: If you make the Readiness probe too smart (e.g., checking if it's blacklisted), a temporary cluster-wide load spike could make all BEs "Unready" at once. This would cause Kubernetes to drop the Service IP, potentially breaking your Prometheus scraping and external monitoring exactly when you need them. What about 
SELECT 1
? Do not put 
SELECT 1
in a BE probe. If the FE leader is under load or switching, all 50+ BE probes would fail at the same time, triggering a catastrophic cluster-wide restart (a "Readiness Storm"). The 3-Tier Health Hierarchy for StarRocks 4.0.10 Tier | Type | Metric/Logic | Purpose Tier 1: Infrastructure | Liveness | 
/api/health
+ 
exec_state_report_queue_count < 1000
| Restart the pod only if the BE-to-FE communication is deadlocked. Tier 2: Application | Query Queues | 
query_queue_cpu_used_permille_limit = 800
| Prevention. FE sees the high CPU and stops sending new queries before the worker pool jams. Tier 3: Watchdog | External | 
SHOW PROC '/current_queries'
| An external script that kills queries running longer than 5 minutes with 
LIKE
patterns, rather than killing the whole pod. Summary Recommendation Use the Scripted Liveness Probe above. It addresses the specific failure mode where the BE HTTP server is "up" but the internal execution engine has stopped talking to the FE. Simultaneously, enable Query Queues on the FE to prevent the worker pool saturation from happening in the first place. References • administration/management/monitoring/metrics.md • administration/management/resource_management/query_queues.md
n
@Rocky but during the time when the incident happened for me, the value of metric 
starrocks_be_exec_state_report_queue_count
never went above 12, so it doesnt seem to be right metric for liveness probe
r
If 
starrocks_be_exec_state_report_queue_count
stayed low, it confirms that the reporting threads (the messengers) were idle because the execution threads (the workers) were completely stuck and never finished any work to report. In StarRocks 4.0.10 (Pipeline Engine), a worker pool saturation from heavy 
LIKE
patterns or complex regex doesn't backlog the "report queue"; it backlogs the Pipeline Driver Queue. 1. The Correct Liveness Metrics for Execution Saturation To detect a BE that is "alive" but "executing nothing," you must look at the Pipeline Engine status. Here are the exact metrics from 
<http://localhost:8040/metrics>
you should use: * 
starrocks_be_pipe_driver_queue_len
: This is the number of tasks ready to run that are waiting for a CPU core. If this is high while CPU is 100%, your workers are saturated. * 
starrocks_be_cpu_used_permille
: 1000 = 100% CPU. In your incident, this was likely 990-1000. * 
starrocks_be_pipe_poller_block_queue_len
: If this is high, tasks are stuck waiting for "something" (usually I/O or network), which can also cause RPC timeouts. 2. The Recommended "Deep" Liveness Probe This probe triggers a restart only if the BE is pinned at max CPU AND has a backlog of tasks that aren't moving for a sustained period. This prevents a "Broken pipe" from turning into a permanent "Zombie BE."
Copy code
yaml
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      # 1. Verify process health
      curl -sSf <http://localhost:8040/api/health> || exit 1

      # 2. Extract metrics
      METRICS=$(curl -s <http://localhost:8040/metrics>)
      CPU=$(echo "$METRICS" | grep 'starrocks_be_cpu_used_permille' | awk '{print $2}')
      QUEUE=$(echo "$METRICS" | grep 'starrocks_be_pipe_driver_queue_len' | awk '{print $2}')

      # 3. Logic: If CPU > 98% AND more than 50 tasks are queued, 
      # the execution engine is saturated/stalled.
      if [ "$CPU" -gt 980 ] && [ "$QUEUE" -gt 50 ]; then
        echo "BE Execution Saturated: CPU ${CPU}m, Queue ${QUEUE}"
        exit 1
      fi
  initialDelaySeconds: 60
  periodSeconds: 30
  failureThreshold: 6 # 3 minutes total of saturation before restarting
  timeoutSeconds: 10
3. Why this probe works for your incident 1. Detection: In your 
LIKE
-pattern incident, the CPU was 100%, and the 
pipe_driver_queue_len
would have skyrocketed as the FE kept sending more fragments that the BE couldn't start. 2. Conservative Restart: By checking both CPU and Queue, we ensure we don't kill a BE just because it's "busy." We only kill it if it's busy and falling behind. 3. Grace Period: The 
failureThreshold: 6
(3 minutes) is vital. It gives the StarRocks Query Timeout (default 5 mins) a chance to kill the bad query first. A pod restart should always be the absolute last resort. 4. Important: The "Broken Pipe" Fix The "Broken pipe" and blacklisting occurred because the BE's internal RPC server (brpc) was so overwhelmed it couldn't even accept the FE's heartbeat connection. * Prevention: You should set 
query_queue_cpu_used_permille_limit = 800
in your FE configuration. * Result: When the BE hits 80% CPU, the FE will automatically stop sending new fragments to that BE. This keeps the BE's CPU at 80-90%, leaving enough headroom for the RPC server to respond to heartbeats, preventing the "Broken pipe" and the blacklist entirely. Summary of checks: * Liveness: Use the script above (CPU + 
pipe_driver_queue_len
). * Readiness: Keep it simple (
/api/health
). Do not make it deep. * Config: Set 
query_queue_cpu_used_permille_limit
on the FE. This is the "K8s-native-like" way StarRocks handles backpressure. Note on metric names: Depending on the exact sub-version of 4.0.x, some metrics might lose the 
starrocks_be_
prefix if scraped locally. Verify with 
curl localhost:8040/metrics | grep pipe_driver_queue_len
on a running pod to be 100% sure of the string. References
n
@Rocky here putting what happened, Context: production StarRocks 4.0.10, shared-nothing, kube-starrocks-operator on EKS. Cluster: 3 FE + 8 BE. Each BE = 8 CPU, 24 GB mem, 240 GB disk. Data ~400 GB. Incident: cluster stopped processing SELECT queries. FE and BE pods stayed Running, kube liveness (curl /api/health) kept returning 200 OK, so k8s never restarted pods. Metric sum(rate(starrocks_be_query_scan_bytes[1m])) was 0 for ~23 minutes while scan_bytes had been at 1-2 GB/s just before. All queries returned: com.starrocks.sql.analyzer.SemanticException: Backend node not found. backend: [be-X ... alive: true inBlacklist: true] 7 of 8 BEs went inBlacklist=true. FE had marked them blacklisted after thrift RPC failures (many "org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe" on FE), and the blacklist never auto-cleared. Manual FE+BE restart was needed to recover. Trigger was a burst of heavy user queries (LIKE '%a%b%%%%%%%%...' with 100s of wildcards, and cross-tenant JOINs scanning 200M+ rows) that saturated BE CPU to ~99% simultaneously across all 8 BEs. BE process never crashed, never OOMed, no container restart. Just stopped responding to FE fast enough. Metrics we looked at during the freeze: - starrocks_be_query_scan_bytes: dropped to 0 for 23 min - starrocks_be_exec_state_report_queue_count: max 7 across all BEs (baseline 0) - starrocks_be_pipe_drivers: spiked 0 -> 60-117 on 4/8 BEs at bomb start - starrocks_be_pipe_poller_block_queue_len: max 1 (useless) - starrocks_be_pipe_poller_block_queue_len: max 1 (useless) - starrocks_be_pipe_driver_queue_len and starrocks_be_cpu_used_permille: not present in our 4.0.10 metrics scrape - container CPU: pegged at 99% on all 8 BEs at incident start - FE metrics: normal, no crash Ask: 1. What livenessProbe (exec + curl /metrics + threshold) do you recommend for BE and for FE in a shared-nothing StarRocks 4.0.10 cluster on Kubernetes that would specifically detect this "BE alive but FE blacklisted, cluster frozen" state? Include the exact metric names available in 4.0.10 and thresholds you have seen work in prod. 2. What readinessProbe should we add on FE to make FE evict itself from the Service (and stop taking client traffic) when it sees all/most BEs on its own blacklist? Any SQL/HTTP endpoint on FE that safely exposes SHOW BACKENDS / blacklist state for a probe? 3. Should we also tune fe.conf blacklist parameters (black_host_history_sec, black_host_connect_failures_within_time, heartbeat_timeout_second, heartbeat_retry_times) so a transient RPC storm does not leave BEs permanently blacklisted? What values do StarRocks devs recommend for an 8 BE cluster under bursty analytical load? 4. Is there a supported API/SQL to auto-clear BE blacklist entries when the BE has been passing heartbeats for N seconds, without needing DELETE BACKEND BLACKLIST run by an operator? 5. Bonus: which prod-safe combination of query_queue_* variables and ALTER RESOURCE GROUP big_query_* limits would you set on default_wg to fail-fast a query that scans >5M rows or burns >5 min of CPU on a single BE, so a bad user query cannot saturate all BEs and trigger this blacklist cascade again? Please give exact metric names, exact fe.conf keys, exact SQL, and the threshold values you have seen tested in real deployments. Prefer answers that work on 4.0.10 without upgrade.
r
For StarRocks 4.0.10 in a shared-nothing environment on EKS, the "Blacklist Cascade" occurs because the BE's Thrift/bRPC workers are saturated by complex 
LIKE
execution, preventing the FE heartbeats or RPCs from completing within the default timeouts. Here is the production-hardened strategy to detect and prevent this scenario. 1. Robust BE Liveness Probe (Scripted) Since your BEs stayed at 99% CPU but 
/api/health
(which is served by a separate web thread) stayed healthy, you need a probe that detects Execution Saturation. In StarRocks 4.0.10, the metric for the task queue is 
pipe_driver_queue_len
. If your scraper didn't see it, it's likely due to the 
starrocks_be_
prefix being omitted in the local endpoint or a regex mismatch. Recommended BE Liveness Probe:
Copy code
yaml
livenessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      # 1. Base process check
      curl -sSf <http://localhost:8040/api/health> || exit 1
      
      # 2. Check Pipeline Queue. In 4.0.10, the metric is often 'pipe_driver_queue_len' 
      # or 'starrocks_be_pipe_driver_queue_len'. We check both.
      METRICS=$(curl -s <http://localhost:8040/metrics>)
      P_QUEUE=$(echo "$METRICS" | grep -E '^(starrocks_be_)?pipe_driver_queue_len' | awk '{print $2}')
      
      # 3. If the queue is consistently high (> 100), the BE is "zombie" (executing but not finishing)
      if [ -n "$P_QUEUE" ] && [ "$P_QUEUE" -gt 100 ]; then
        echo "BE Pipeline Queue Saturated: $P_QUEUE"
        exit 1
      fi
  initialDelaySeconds: 60
  periodSeconds: 30
  failureThreshold: 6 # 3 minutes total of saturation before K8s restarts the pod
2. FE Readiness Probe (Eviction Strategy) To prevent the FE from receiving traffic when it has blacklisted its own backends, you can use a script that queries the FE's internal proc information. Recommended FE Readiness Probe:
Copy code
yaml
readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - |
      # Check if more than 50% of backends are blacklisted
      TOTAL_BE=$(curl -s -u admin: <http://localhost:8030/api/show_proc?path=/backends> | grep -c "\[")
      BLACK_BE=$(curl -s -u admin: <http://localhost:8030/api/show_proc?path=/backends> | grep -c "true") # Greps 'InBlacklist' column
      
      # Evict FE if most BEs are gone
      if [ "$BLACK_BE" -ge "$((TOTAL_BE / 2))" ]; then
        exit 1
      fi
3. Tuning 
fe.conf
Blacklist Parameters The default parameters are often too aggressive for bursty EKS environments. In an 8 BE cluster, a single "query bomb" can trigger a 10s RPC block, causing a permanent blacklist. Add/Modify in `fe.conf`: Parameter | Recommended Value | Reason 
black_host_history_sec
| 
60
| Reduces the "memory" of failures so BEs can rejoin faster. 
black_host_connect_failures_within_time
| 
10
| Default is 5. Increasing this gives the BE more "chances" during an RPC storm. 
heartbeat_timeout_second
| 
10
| Increase from 5s to 10s to account for CPU-steal on EKS during high load. 
heartbeat_retry_times
| 
5
| Allows more transient failures before marking the node as dead. 4. Auto-clearing the Blacklist There is no "clear all" SQL. However, you can force the FE to re-evaluate all nodes by reducing the 
black_host_history_sec
. * Manual Clear: 
DELETE BACKEND BLACKLIST 10001, 10002...;
* Recommended Automation: Use an external CronJob or sidecar that runs the following every minute:
Copy code
sql
-- This doesn't exist as a single command, but you can automate the output of:
SELECT group_concat(BackendId) FROM information_schema.be_backends WHERE InBlacklist = 'true';
-- Then run DELETE BACKEND BLACKLIST <ids>;
5. Bonus: Fail-Fast Protection (Resource Groups) This is the most important step to prevent the incident from recurring. You must constrain the 
default_wg
or create a 
big_query
group. Run these SQL commands: ```sql -- 1. Enable Query Queuing (FE level) ADMIN SET FRONTEND CONFIG ("query_queue_concurrency_limit" = "20");
ADMIN SET FRONTEND CONFIG ("query_queue_cpu_used_permille_limit" = "800"); -- 2. Create a "Bad Query" jail or update default limits ALTER RESOURCE GROUP default_wg SET ( "big_query_cpu_second_limit" = "300", -- Kill any query using > 5 min of CPU "big_query_scan_rows_limit" = "5000000", -- Kill any query scanning > 5M rows "big_query_mem_limit" = "8589934592", -- Kill any query using > 8GB RAM per BE "concurrency_limit" = "10" -- Strict limit on concurrent complex queries ); -- 3. Enable SQL Blacklist for the specific regex (e.g., the bad LIKE pattern) ADMIN SET FRONTEND CONFIG ("enable_sql_blacklist" = "true"); ADD SQLBLACKLIST ".*LIKE\s+'%.*%.*%.*%.*'.*"; ``` Why this combination works: * 
big_query_cpu_second_limit
: Specifically stops the "infinite 
LIKE
loop" from pinning the CPU for 20+ minutes. * 
query_queue_cpu_used_permille_limit
: Prevents the FE from sending any new queries to a BE that is already at 80% CPU, leaving the remaining 20% for heartbeats and RPCs, thus avoiding the "Broken Pipe." References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md
7 Views
Open in Slack
PreviousNext
Powered by