@Rocky Does the starrocks_be_cpu metric represent CPU usage at the node level or the pod level?

The metric

starrocks_be_cpu

represents CPU usage at the node level (the host system), rather than the pod or process level. Key Details * Source: This metric is derived directly from the

/proc/stat

file on the system where the Backend (BE) is running. In a Linux environment,

/proc/stat

provides aggregate CPU statistics for the entire machine. * Behavior in Kubernetes: When StarRocks is deployed on Kubernetes,

/proc/stat

inside a container (pod) typically still reflects the statistics of the underlying host node unless specific container isolation features (like

lxcfs

) are used to virtualize

/proc

. Therefore, in a standard K8s setup, this metric will show the CPU usage of the entire node. * Structure: The metric is a counter that includes various CPU modes as labels (e.g.,

mode="user"

,

mode="system"

,

mode="idle"

,

mode="iowait"

). How to calculate CPU Utilization To calculate the actual CPU utilization percentage from this metric in Prometheus, you typically use the

idle

mode:

promql
(1 - sum(rate(starrocks_be_cpu{mode="idle"}[5m])) by (instance) / sum(rate(starrocks_be_cpu[5m])) by (instance)) * 100

Pod-Level Alternatives If you specifically need to monitor pod-level CPU usage (i.e., how much the individual StarRocks BE container is consuming relative to its limits), you should use Kubernetes-native metrics provided by

cAdvisor

or

kubelet

instead of StarRocks internal metrics: *

container_cpu_usage_seconds_total

: The total CPU time consumed by the container. *

container_spec_cpu_quota

: The CPU limit assigned to the pod. Summary Metric | Level | Source

starrocks_be_cpu

| Node (Host) |

/proc/stat

container_cpu_usage_seconds_total

| Pod (Container) | cgroups (via Prometheus/cAdvisor) For more details on monitoring StarRocks, you can refer to the official Monitoring and Alerting documentation. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/proc_profile.md • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md

@Rocky If a query is killed, are the fragments that have already been dispatched to other CN nodes also cancelled?

Yes, when a query is killed in StarRocks, all fragments dispatched to Compute Nodes (CNs) or Backends (BEs) are also cancelled. How it works: 1. Trigger: A query can be killed for several reasons, including a manual

KILL QUERY

command, a query timeout (controlled by

query_timeout

), or the client connection being closed/lost. 2. Coordination: The Front End (FE) that is coordinating the query (the Coordinator) detects the kill event or timeout. 3. Cancellation Signal: The FE immediately sends an asynchronous RPC (Remote Procedure Call) cancellation signal to every BE or CN node that was assigned a fragment of that query. 4. Local Execution Halt: Upon receiving this signal, each CN/BE node stops the execution of the relevant Plan Fragment Instances. This includes: * Stopping data scanning and processing. * Releasing memory buffers allocated for the query. * Halting network data exchange between nodes for that specific query ID. Why this happens: This mechanism ensures that a killed query does not continue to consume CPU, memory, or I/O resources on worker nodes after the user or the system has decided to terminate it. Verifying Cancellation: If you want to verify that fragments are being cleaned up, you can monitor the

plan_fragment_count

metric on your CN/BE nodes. When a large query is killed, you should see this count decrease as the fragment instances are terminated and removed from the execution queue and finalized. References • administration/management/resource_management/Query_management.md • sql-reference/sql-statements/cluster-management/nodes_processes/KILL.md • best_practices/query_tuning/query_planning.md • administration/management/monitoring/metrics.md References • best_practices/query_tuning/query_planning.md • administration/management/FE_configuration.md • using_starrocks/Cost_based_optimizer.md • administration/management/monitoring/metrics.md • administration/management/resource_management/Query_management.md • administration/management/compaction.md • administration/user_privs/authorization/privilege_faq.md • sql-reference/sql-statements/cbo_stats/KILL_ANALYZE.md