@Rocky in the monitoring part of the docs, some parts says about template but I don't actually see any templates like the json for grafana dashboard. Can you point me to docs that really includes the actual metrics and the actual template that is readily available? https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data/#starlet-dashboard

The Grafana dashboard templates (JSON files) are not hosted directly within the documentation's markdown text but are provided as downloadable links in the Monitor and Alert guide. 1. Downloadable Grafana Templates You can find the latest JSON templates at these direct links: * Dashboard template for All Architecture * Dashboard template for Shared-data Cluster - General * Dashboard template for Shared-data Cluster - Starlet 2. Metric Lists For the actual metrics used in these dashboards, the documentation categorizes them as follows: * General Metrics (FE/BE): General Monitoring Metrics – Lists standard CPU, memory, query, and tablet metrics. * Shared-data Specific Metrics: Monitoring Metrics for Shared-data Clusters – Detailed descriptions of metrics for the Object Storage Layer, Starlet, and Cache usage. How to use them 1. Download the JSON file corresponding to your cluster architecture. 2. In your Grafana Web UI, go to Dashboards -> New -> Import. 3. Upload the JSON file. 4. Select your Prometheus data source (configured to scrape StarRocks'

/metrics

endpoint). The documentation detailing this setup process can be found here: Monitor and Alert with Prometheus and Grafana. References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/metrics.md • loading/loading_introduction/troubleshooting_loading.md

@Rocky what is the meaning of the following modes in starrocks_be_cpu and how can I use them? guest guest_nice idle iowait irq nice soft_irq steal system user

The metric

starrocks_be_cpu

(and its counterpart

starrocks_fe_cpu

) provides a breakdown of CPU time spent by the operating system, categorized by modes. These are standard Linux kernel CPU states typically sourced from

/proc/stat

. Meaning of each mode Mode | Meaning user | Time spent running un-niced user-level processes. In a StarRocks node, this is where the core computation for queries and data processing happens. system | Time spent running kernel-level processes (system calls). High values often indicate heavy I/O or networking overhead. idle | Time spent in an idle state. If this is consistently low (e.g., < 10%), your BE node is CPU-bound. iowait | Time spent waiting for I/O to complete. High values suggest your disks or storage backend (S3/HDFS) are a bottleneck. nice | Time spent running user-level processes that have a positive

nice

value (lower priority). irq | Time spent servicing hardware interrupts. soft_irq | Time spent servicing software interrupts (common in high network traffic). steal | Time "stolen" by the hypervisor for other virtual machines. Only relevant if you are running StarRocks on VMs (like AWS EC2). High values mean your VM host is oversubscribed. guest | Time spent running a virtual CPU for guest operating systems. Usually 0 on a StarRocks node. guest_nice | Time spent running a "niced" guest CPU. Usually 0. How to use these metrics These metrics are counters (cumulative time). To get a meaningful "percentage" usage in Prometheus/Grafana, you use the

irate

or

rate

function. 1. Calculate Total CPU Usage % (Excluding Idle) To see how busy your BE nodes are, subtract the idle rate from the total:

promql
1 - (irate(starrocks_be_cpu{mode="idle"}[5m]) / ignoring(mode) group_left sum by (instance)(irate(starrocks_be_cpu[5m])))

2. Troubleshooting Bottlenecks * High `user`: Normal for heavy query loads. If it hits 100%, consider adding more BE nodes or optimizing queries. * High `iowait`: Check disk performance. For shared-data clusters, this may indicate issues with the local cache disk or object storage latency. * High `system`: Can indicate excessive context switching or kernel overhead. Check if

transparent_hugepages

is disabled as recommended in the StarRocks installation guide. * High `steal`: If you are on a cloud provider (AWS/GCP/Azure), this means your VM is being throttled or the physical host is overloaded. You may need to upgrade to a dedicated instance or a larger instance type. Where to find this in the Docs While these are standard Linux modes, StarRocks references their general usage in the Monitor and Alert section, specifically focusing on the BE CPU Idle metric as the primary health indicator. References • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md