@Rocky Recommend a comprehensive set of resource group limits I should impose on all users to ensure queries do not overwhelm the system and enough resources are always available for ingestion

To ensure system stability while maintaining high ingestion performance, I recommend implementing a multi-tiered Resource Group strategy. This approach separates resource-intensive ETL tasks from interactive queries and applies "safety rails" to prevent runaway queries from crashing the system. Recommended Resource Group Architecture You should create at least three distinct resource groups based on workload types: 1. Interactive Resource Group (High Priority) For short, latency-sensitive dashboard queries and small ad-hoc lookups. *

cpu_weight

: Set high (e.g.,

10

) to prioritize these over background tasks. *

mem_limit

: Moderate (e.g.,

0.3

or 30%). *

concurrency_limit

: High (e.g.,

100+

) to allow many users to work simultaneously. * Big Query Limits: Set strict limits here to ensure "big" queries don't accidentally run in the interactive lane. * `big_query_cpu_second_limit`: 20–30s. * `big_query_mem_limit`: 2–4GB. 2. Ad-hoc/Analytical Resource Group (Normal Priority) For data analysts running complex joins or large aggregations. *

cpu_weight

: Moderate (e.g.,

5

). *

mem_limit

: High (e.g.,

0.5

or 50%) since these queries are memory-heavy. *

concurrency_limit

: Low (e.g.,

5–10

) to prevent too many large scans from saturating disk I/O. * Big Query Limits: More generous, but still present. * `big_query_cpu_second_limit`: 300s (5 mins). * `big_query_scan_rows_limit`: 1,000,000,000. 3. ETL/Ingestion Resource Group (Background Priority) Specifically for

INSERT INTO

and

Broker Load

jobs. *

cpu_weight

: Low (e.g.,

2

) so they don't starve interactive users during peak hours. *

mem_limit

: Moderate (e.g.,

0.2

or 20%). *

query_type

: Bind this group to

('insert')

via classifiers. * Note: This ensures that heavy transformation during

INSERT INTO SELECT

doesn't overwhelm the system. ────────── Critical "Safety Rail" Limits Apply these limits to all resource groups to prevent a single query from causing a Backend (BE) to OOM (Out of Memory) or hang. Parameter | Recommended Value | Why?

big_query_mem_limit

| ~10-20% of BE RAM | Kills any single query that consumes too much memory on a single node.

big_query_cpu_second_limit

| 600 (10 mins) | Prevents "zombie" queries with infinite loops or massive cross-joins from hogging cores.

concurrency_limit

| Varies by group | Essential to prevent a "thundering herd" of requests from exhausting the query thread pool.

spill_mem_limit_threshold

|

0.8

(80%) | Triggers [Intermediate Result Spilling](https://docs.starrocks.io/docs/administration/management/resource_management/spill_to_disk/) to disk when memory is tight, preventing OOM. Important Implementation Details 1. Handling Stream Load & Routine Load: As of v3.3.5, Stream Load and Routine Load are NOT managed by Resource Groups. To limit their impact, you must adjust global BE configurations: * `load_process_max_memory_limit_percent`: Default is 30%. Lower this if ingestion is starving queries. * `max_routine_load_task_num_per_be`: Limits the number of concurrent Kafka consumption tasks. 2. Enable Query Queuing: Starting in v3.1, you can enable

enable_group_level_query_queue = true

. This is better than a hard

concurrency_limit

because it puts excess queries in a queue instead of failing them with an error. 3. Hard CPU Limits (v3.3.5+): If you have a strict requirement that Ingestion must always have 4 dedicated cores, use

exclusive_cpu_cores

. This is a "hard limit" that prevents other groups from touching those cores, even if they are idle. Example SQL for Ingestion Group

sql
CREATE RESOURCE GROUP etl_group
TO (query_type IN ('insert'))
WITH (
    "cpu_weight" = "2",
    "mem_limit" = "0.2",
    "concurrency_limit" = "5",
    "big_query_mem_limit" = "10737418240" -- 10GB safety rail
);

References

@Rocky for starrocks_be_cpu metrics, what processes fall under "user" mode and what fall under "system" mode. For refenrece im using shared nothign primary key table, ingestion via stream load with merge commit enabled.

For the

starrocks_be_cpu

metric, "user" and "system" do not refer to separate OS processes, but rather to the execution modes of the single

starrocks_be

process. In your specific environment (Shared-nothing, Primary Key table, Stream Load with Merge Commit), here is how the workload is categorized: 1. User Mode CPU (

user

) This represents time spent executing StarRocks' own application code. For your Primary Key and Stream Load setup, the following tasks are the primary consumers: * Primary Key Index Management: Computing hashes for primary keys and performing lookups/updates in the Persistent Index. This is heavily CPU-intensive in user-space. * Transaction Apply (Upsert Logic): The logic that compares incoming Stream Load data with existing row versions to determine which rows to update or delete (

transaction_apply_worker

). * Data Parsing & Vectorization: Parsing incoming CSV/JSON data from Stream Load and executing the vectorized engine logic (filtering, expressions, or sorting). * Compaction: Background threads merging rowsets to maintain query performance. While this involves I/O, the sorting and merging of data chunks happen in user mode. * Merge Commit Logic: The internal coordination of buffering multiple small Stream Load requests and merging them into a single memory buffer before flushing. 2. System Mode CPU (

system

) This represents time spent by the Linux kernel on behalf of the BE process. High system CPU in StarRocks usually points to infrastructure or resource contention issues: * Network I/O (RPCs): Handling the high volume of incoming HTTP Stream Load requests and the internal brpc (Baidu RPC) communication between BEs and the FE for transaction coordination. * Disk I/O (Syscalls): The actual

write

,

fsync

, and

read

calls to the local disk. In a shared-nothing setup, high system CPU can occur if you have a "small file problem" (too many small commits), which

Merge Commit

is specifically designed to mitigate. * Context Switching: If you have too many active threads (e.g.,

transaction_apply_worker_count

or

compact_threads

set too high), the kernel spends excessive time switching between them. * Memory Allocation: Large or frequent allocations via

jemalloc

that require kernel intervention (like

mmap

or

munmap

) when the BE is near its memory limit. Recommended Observability If you see System CPU spiking above 10-15%, it typically indicates a bottleneck outside of StarRocks' logic: * Check Disk Latency: Use

iostat -x 1

to see if the disks are saturated. * Check RPC Health: High system CPU often correlates with network congestion or extremely high concurrency of small loads. * Audit Merge Commit: Ensure

merge_commit_interval_ms

is high enough (e.g., 1000ms+) so you aren't overwhelming the kernel with thousands of tiny file IOPS. Related Metrics for Investigation * `starrocks_be_update_rowset_commit_apply_total`: Monitors the throughput of PK table commits. * `starrocks_be_tablet_cumulative_max_compaction_score`: If high, the BE is working hard on background merges (User CPU). * `starrocks_be_resource_group_cpu_use_ratio`: If using resource groups, this shows how much of the "User" time is being consumed by specific workloads. References • administration/management/monitoring/metrics.md • best_practices/primarykey_table.md • loading/loading_introduction/troubleshooting_loading.md • administration/management/monitoring/Monitor_and_Alert.md References

@Rocky

starrocks_be_resource_group_cpu_use_ratio

How do I interpret this metric? sum(starrocks_be_resource_group_cpu_use_ratio{starrocks_cluster="core-services"}) by (name) Shows my default_wg having value 8.26 while resource group ml_zeppelin having value 3.58

To interpret the metric

starrocks_be_resource_group_cpu_use_ratio

in your Prometheus query, you need to understand its scale and what it represents relative to your cluster size. 1. What the value represents This metric measures the CPU time used by a specific resource group relative to the total CPU time consumed by ALL resource groups on a Backend (BE) node. * On a single BE: The sum of this ratio across all resource groups (including

default_wg

) will always equal 1.0 (or 100% of the CPU currently managed by the resource group scheduler). * In your

sum(...) by (name)

query: Because you are summing this ratio across all BEs in your cluster, the value represents the aggregate "node-equivalents" of CPU priority. 2. Interpreting your specific numbers You have a cluster (let's assume it has 12 BE nodes for this example): *

default_wg

= 8.26 *

ml_zeppelin

= 3.58 * Total = 11.84 (This is very close to 12, which suggests your cluster's managed CPU is nearly fully utilized). The Breakdown: *

default_wg

is consuming ~70% (8.26 / 11.84) of the cluster's managed CPU resources. *

ml_zeppelin

is consuming ~30% (3.58 / 11.84) of the cluster's managed CPU resources. If the "Total" sum is significantly lower than your total number of BEs, it means the resource-group-managed threads (mostly the Pipeline engine used for queries) are idle or the system is not under full load. 3. Key Distinctions * Relative, not Absolute: This metric does not tell you the literal number of CPU cores being used. Instead, it tells you the distribution of work. For absolute core usage, look for the metric

resource_group_inuse_cpu_cores

(if available in your version). * Only Managed CPU: This ratio only covers tasks governed by resource groups (SQL queries,

INSERT INTO...SELECT

). It typically does not include background system tasks like Compaction, Checksum, or Tablet Meta management, which are not currently part of the resource group "pie." * Workload Balance: If you see

default_wg

consistently much higher than your custom groups, it indicates that most of your workload is not yet classified into specific groups, or your

cpu_weight

settings are allowing the default group to dominate. Recommendation for Better Insight To see the percentage of cluster-wide managed CPU each group is using, use this PromQL:

promql
sum(starrocks_be_resource_group_cpu_use_ratio) by (name) 
/ 
ignoring(name) group_left sum(starrocks_be_resource_group_cpu_use_ratio)

This will give you a value between 0 and 1 for each group, showing its share of the current query load. References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md