http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> im currently setting up resource gr...
# questions-and-troubleshooting
r
@Rocky im currently setting up resource groups in my cluster (starrocks 4.0.8, shared-data mode). the bulk of our traffic (by far) comes from backend jobs, where we can tolerate some latency. we also get traffic from our frontend, which we need to make sure arent stuck waiting on the job traffic. what cluster metrics should i look at to determine the appropriate limits for each of the two resource groups i intend to create?
r
Setting up resource groups in StarRocks 4.0.8 (shared-data mode) is an excellent way to ensure your frontend users stay snappy while your backend jobs crunch through data. Since you are in shared-data mode, your "BE" nodes act as the compute layer, and you want to prevent backend jobs from saturating their CPU, memory, and local data cache. To determine the right limits, you should look at historical performance data and real-time resource group metrics. 1. Key Metrics to Analyze (Historical) Before creating the groups, query your Audit Log (stored in 
starrocks_audit_db*.starrocks_audit_tbl*
) or check the 
information_schema
. This will help you distinguish the resource footprints of your two workloads. * CPU Consumption (
cpuCostNs
): Sum up the total CPU time per user/workload. _ _Goal:* Determine if the frontend needs a guaranteed number of cores (
exclusive_cpu_cores
) versus the backend which can use a weight-based share (
cpu_weight
). * Peak Memory Usage (
memCostBytes
): Identify the 
MAX(memCostBytes)
for your largest backend jobs. _ _Goal:* Set a 
big_query_mem_limit
for the backend group so one "rogue" job doesn't OOM a node. * Query Concurrency: Calculate the number of overlapping queries at peak times for both frontend and backend. _ _Goal:* Set 
concurrency_limit
to prevent backend jobs from filling up the internal query queue. * Scan Volume (
scanRows
/ 
scanBytes
): Backend jobs usually scan massive amounts of data. _ _Goal:* Use these to set 
big_query_scan_rows_limit
to catch runaway jobs. 2. Implementation Strategy for Your Use Case Based on your description, here is a recommended configuration strategy: * For the Frontend (Low Latency): * Use `exclusive_cpu_cores`: This provides hard isolation. If you have a 32-core BE, you might reserve 4–8 cores exclusively for the frontend. These cores will never be "stolen" by backend jobs. * High `mem_limit`: Ensure it has enough headroom (e.g., 
0.4
or 40%) so it never fails due to memory exhaustion. * For Backend Jobs (High Throughput): * Use `cpu_weight`: Instead of hard-reserving cores, give them a weight. If the frontend is idle, the backend can use all available CPU. If the frontend is busy, the backend will be throttled. * Set `concurrency_limit`: Since backend jobs are "bulky," limit how many can run at once (e.g., 10–20) to ensure the system remains responsive. * Set 
big_query_...
limits: Backend jobs are prone to "exploding." Set limits on CPU seconds and memory per query to kill outliers. 3. Monitoring Metrics (Live) Once your resource groups are active, monitor these specific BE metrics (typically via Prometheus/Grafana) to see if your limits are working: Metric | Why it matters for you 
resource_group_inuse_cpu_cores
| Shows if your frontend is actually hitting its reserved limit. 
resource_group_mem_inuse_bytes
| Helps you see if the backend is getting close to its 
mem_limit
. 
resource_group_concurrency_overflow_count
| If this is rising for the backend, your jobs are being queued (which is fine for latency-tolerant jobs!). 
fslib_cache_hit_ratio
| Critical for Shared-Data. If backend jobs are scanning too much data, they might evict the frontend's "hot" data from the local BE cache. If this drops for the frontend, consider increasing the BE local disk space. Summary Query to Get Started Run this on your cluster to see the current "split" between your frontend and backend users (replace 
'frontend_user'
and 
'job_user'
with your actual identifiers):
Copy code
sql
SELECT 
    user,
    COUNT(*) as query_count,
    AVG(cpuCostNs) / 1e9 as avg_cpu_sec,
    MAX(memCostBytes) / 1024 / 1024 as max_mem_mb,
    PERCENTILE_APPROX(queryTimeMs, 0.99) as p99_latency_ms
FROM starrocks_audit_db__.starrocks_audit_tbl__
WHERE timestamp >= DATE_SUB(NOW(), INTERVAL 1 DAY)
GROUP BY user;
Rocky's Pro Tip: In StarRocks 4.0, you can also use 
mem_pool
to group multiple resource groups together under one shared memory ceiling. This is great if you have different "types" of backend jobs but want them to share a single block of memory. References • best_practices/audit_log_resource_group.md • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md
r
@Rocky shouldnt most metrics be related to the specifics of my cluster setup (number of nodes, size of nodes, allocated memory, etc)? moreso than the query speciifcs weve see so far?
r
You are absolutely correct. The query-level metrics tell you the demand, but your cluster's hardware specs define the supply. In StarRocks, resource group limits are strictly relative to the capacity of your individual BE (Compute) nodes. Since you are on Shared-Data mode (v4.0.8), you are managing a decoupled compute layer. Here are the specific cluster-level metrics and hardware specs you should look at to anchor your limits: 1. Identify the "Total Supply" (The Denominator) Before setting percentages, you need the hard ceiling of your hardware. * CPU Cores per Node: Check 
min_be_cpu_cores
(the smallest CPU count among your BEs). _ _Constraint:* The sum of all 
exclusive_cpu_cores
across all groups must be less than 
min_be_cpu_cores - 1
. * Memory per Node: Check the 
mem_limit
in your 
be.conf
(defaults to 90% of total RAM). _ Constraint: Your resource group 
mem_limit
is a percentage of _this* allocated memory, not the total physical RAM. * Local Cache Disk Size: In shared-data mode, each BE has a local disk cache (configured via 
storage_root_path
). If this is small (e.g., 500GB) and your backend jobs scan 2TB, they will "evict" your frontend's hot data, causing latency spikes regardless of CPU limits. 2. Cluster Metrics to Look at (The Pressure Test) Look at these BE-level metrics in Prometheus/Grafana over a 24-hour cycle to see your "Current Headroom": Metric Name | What it tells you 
starrocks_be_cpu_idle
| If this is consistently < 20% during backend jobs, you must use 
exclusive_cpu_cores
for the frontend to prevent starvation. 
starrocks_be_mem_usage
| Shows how close you are to the process limit. If it's jagged/spiky, your backend jobs need a strict 
big_query_mem_limit
. 
fslib_cache_hit_ratio
| Critical for Shared-Data. If this drops when jobs run, the backend is "polluting" the cache. You might need to increase local disk or use 
datacache.enable = false
for certain backend tables. 
fslib_read_throughput
| Shows the S3/Object Storage bandwidth. If the backend saturates this, frontend queries will wait on IO, not CPU. 3. Calculating the Limits (The Formula) To bridge your hardware specs with your workload, use this logic: A. Frontend Resource Group (The "Priority Lane") * Metric to check: Peak 
BEInUseCpuCores
for frontend users during high traffic (via 
SHOW USAGE RESOURCE GROUPS
). * Limit: Set 
exclusive_cpu_cores
to slightly above that peak. _ _Example:* If frontend users typically peak at 3.5 cores on a 16-core machine, set 
exclusive_cpu_cores = 4
. This "slices off" 4 cores that backend jobs can never touch. B. Backend Resource Group (The "Throughput Lane") * Metric to check: Average 
BEInUseMemBytes
for your batch jobs. * Limit: Use 
cpu_weight
instead of exclusive cores. _ _Logic:* If the frontend is idle, the backend can use all 16 cores. If the frontend is active, the backend is throttled to the remaining 12. * Limit: Set a strict 
mem_limit
(e.g., 
0.5
or 50%). Since jobs can tolerate latency, it is better for a job to wait for memory than to crash the BE node. 4. Shared-Data Specific Tip: The "Cache" Factor In 4.0.8, backend jobs often perform heavy "Cold Reads" from S3. These reads use a specific thread pool. If you see high 
starrocks_be_resource_group_cpu_use_ratio
but low frontend performance, check the S3 GetObject Latency. If S3 is the bottleneck, increasing CPU limits won't help; you'll instead need to limit the 
concurrency_limit
of the backend group to reduce the IO pressure on the network interface. Recommendation: Start by running 
SHOW BACKENDS\G
to see your actual 
MemLimit
and 
CpuCores
across the cluster, then apply the "slice" for the frontend based on the 
exclusive_cpu_cores < min_be_cpu_cores - 1
rule. References
r
@Rocky 
SHOW BACKENDS\G
is empty is that expected?
r
Yes, that is expected in a shared-data mode cluster. In StarRocks shared-data mode, the nodes that perform the computation are technically registered as Compute Nodes (CN) rather than traditional Backends (BE). While they run the same underlying software, they are "stateless" in terms of data ownership (since the data lives on S3/OSS/HDFS). The Command to Use To see your nodes in shared-data mode, you should use:
Copy code
sql
SHOW COMPUTE NODES\G
Why 
SHOW BACKENDS
is empty: 1. Shared-Nothing vs. Shared-Data: In a "Shared-Nothing" cluster (the traditional mode), you add nodes using 
ALTER SYSTEM ADD BACKEND
. These show up in 
SHOW BACKENDS
. 2. Shared-Data Architecture: In "Shared-Data" clusters, you add nodes using 
ALTER SYSTEM ADD COMPUTE NODE
. These nodes show up in 
SHOW COMPUTE NODES
. What to check in the output: When you run 
SHOW COMPUTE NODES\G
, look for these specific fields to help with your resource group setup: * 
CpuCores
: This confirms the actual number of cores the FE (Frontend) thinks the node has. Use this as your baseline for 
exclusive_cpu_cores
. * 
MemLimit
: This is the total memory StarRocks is allowed to use on that node. Your resource group 
mem_limit
(percentage) will be calculated based on this number. * 
Alive
: Ensure this is 
true
. If the nodes aren't alive, your resource group configurations won't be applied to them. If 
SHOW COMPUTE NODES
is also empty: It means your compute nodes haven't been successfully joined to the Frontend yet. You would need to run:
Copy code
sql
ALTER SYSTEM ADD COMPUTE NODE "IP:9050"; -- Replace with your node's IP and Heartbeat Port
Rocky's Pro Tip: Even though they are called "Compute Nodes," you will still see references to "BE" in the configuration files (
be.conf
) and in many metric names (like 
starrocks_be_cpu_usage
). Just remember that for cluster management SQL commands, CN is the keyword for Shared-Data and BE is for Shared-Nothing. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • deployment/deploy_manually.md
r
@Rocky i have 3 compute nodes, all with approximately the same stats below:
Copy code
Version: 4.0.8-2090bfc
             CpuCores: 16
             MemLimit: 91.800GB
    NumRunningQueries: 6
           MemUsedPct: 52.22 %
           CpuUsedPct: 8.3 %
     DataCacheMetrics: Status: Normal, DiskUsage: 3.1TB/3.1TB, MemUsage: 18.3GB/18.3GB
r
Based on your node specifications (16 cores, 91.8GB MemLimit, and 3.1TB local cache), you have a solid foundation. Since your backend jobs are latency-tolerant and your frontend is the priority, I recommend a "Hard/Soft" hybrid split. In StarRocks, this is achieved by using 
exclusive_cpu_cores
for your frontend (to guarantee a lane) and 
cpu_weight
for your backend (to allow it to scale up when the cluster is idle). Recommended Resource Group Configuration 1. The Frontend Group (The "VIP" Lane) This group reserves cores that the backend can never touch, ensuring zero CPU contention. * CPU: 
exclusive_cpu_cores = 4
(25% of your 16 cores). This ensures the frontend always has 4 physical cores available per node. * Memory: 
mem_limit = 25%
(approx 23GB). * Concurrency: High limit to avoid queuing users.
Copy code
sql
CREATE RESOURCE GROUP frontend_rg
TO (role='frontend_role') -- or specific users
WITH (
    "exclusive_cpu_cores" = "4", 
    "mem_limit" = "25%",
    "concurrency_limit" = "100"
);
2. The Backend Group (The "Throughput" Lane) This group uses a weight. It can use all 16 cores if the frontend is silent, but if the frontend is busy, it is automatically throttled down to the remaining 12 cores. * CPU: 
cpu_weight = 10
(Any value is fine, but weight-based means it's a "Shared" group). * Memory: 
mem_limit = 60%
(approx 55GB). This allows large jobs to run without starving the system's overhead. * Big Query Protection: Crucial for backend jobs. You should kill jobs that exceed reasonable limits.
Copy code
sql
CREATE RESOURCE GROUP backend_rg
TO (user='job_user')
WITH (
    "cpu_weight" = "10",
    "mem_limit" = "60%",
    "concurrency_limit" = "10", -- Keep this low so jobs don't overwhelm the I/O
    "big_query_mem_limit" = "42949672960", -- 40GB max per query
    "big_query_cpu_second_limit" = "3600", -- 1 hour max CPU time
    "big_query_scan_rows_limit" = "1000000000" -- 1 billion rows max
);
────────── Why these specific metrics matter for your 4.0.8 Setup: 1. CPU (16 Cores): Since you have 16 cores, setting 
exclusive_cpu_cores = 4
leaves 12 cores for the OS and the backend. If your 
CpuUsedPct
is currently only 8.3%, you have a lot of room. If backend jobs spike this to 90%, the frontend's 4 cores will remain untouched, keeping dashboard latency stable. 2. Memory (91.8GB): Your current 
MemUsedPct
is 52.22% (about 48GB). This is already quite high for only 6 queries. * The Trap: If a backend job triggers a massive shuffle/join, it could easily hit the 91.8GB limit and cause the CN node to crash or OOM. * The Fix: Setting a 
big_query_mem_limit
of 40GB (in the backend group) ensures a single bad job cannot tank the node. 3. Data Cache (3.1TB): Your cache is 100% full. This is normal behavior for StarRocks shared-data (it fills the disk and evicts the oldest data). * The Problem: Large backend jobs scanning "cold" data will evict the "hot" data your frontend needs. * The Metric to Watch: 
fslib_cache_hit_ratio
. If this drops when jobs run, your frontend will slow down because it has to go to S3. * Pro Tip: For backend tables that are only read once (batch jobs), you can set the table property 
"datacache.enable" = "false"
. This prevents backend jobs from "polluting" the 3.1TB cache. How to Monitor the Result After applying these, run this query to see if your "Frontend" is hitting its limits:
Copy code
sql
-- Check real-time resource group usage
SELECT * FROM information_schema.resource_group_usage;
If you see 
concurrency_overflow_count
increasing for your backend group, it means your 
concurrency_limit = 10
is doing its job—protecting your frontend by making the jobs wait their turn. References • deployment/shared_data/feature-support-shared-data.md
r
@Rocky what causes 
MemUsedPct
to be so high? theres very little running on these nodes right this second
r
The high 
MemUsedPct
(52.22%) while the cluster is idle is a common observation in StarRocks, especially in Shared-Data mode. It does not mean your queries are currently leaking memory; rather, it's because StarRocks is designed to be "greedy" with memory to ensure high performance for the next query. Here is the breakdown of what is occupying those ~48GB (52.22% of your 91.8GB): 1. The Data Cache (Memory Portion) Your output shows: 
MemUsage: 18.3GB/18.3GB
. * What it is: In Shared-Data mode, StarRocks caches data in both Disk (your 3.1TB) and Memory. * Why it's high: The memory portion of the Data Cache is typically set to 20% of your total BE memory by default (18.3GB is exactly 20% of 91.8GB). * Behavior: This memory is pre-allocated or "sticky." Once it fills up with data from previous queries, it stays at 100% usage. It uses an LRU (Least Recently Used) policy to swap data out, but it never "releases" the memory back to the OS. 2. Metadata (The "Hidden" Cost) StarRocks keeps metadata in memory so it doesn't have to fetch it from the object store during query planning. * Table/Tablet Metadata: Every tablet your node is responsible for has metadata (version info, file locations, etc.) stored in memory. * Column/Index Readers: To speed up scans, StarRocks caches column headers and index structures. If you have many tables or large schemas, this can easily take up 5–10GB. 3. TCMalloc & The "Free List" StarRocks uses 
tcmalloc
as its memory allocator. * The Cache: When a query finishes, 
tcmalloc
often keeps the freed memory in its own internal "freelist" instead of returning it to the Linux kernel immediately. This makes the next memory request much faster. * The Result: From the perspective of the OS (and the 
MemUsedPct
metric), that memory is still "in use" by the StarRocks process, even if no query is currently using it. 4. Other System Caches * Column Pool: Caches reused column objects to reduce object creation overhead. * Compaction: Even if you aren't running queries, the cluster is often performing background compaction to merge small files in the object store. This requires its own memory buffer. ────────── How to see the exact breakdown If you want to see exactly where every megabyte is going, you can access the Memory Tracker directly from your browser or via `curl`:
Copy code
bash
# Replace <CN_IP> with one of your compute node IPs
curl http://<CN_IP>:8040/mem_tracker
What to look for in the output: * `process`: Total memory used by the node. * `DataCache`: Should match your ~18GB. * `Metadata`: Memory used for schemas and tablet info. * `QueryPool`: Should be near zero if idle. * `Tcmalloc`: This will show you how much is being held in the "freelist." Is this a problem? No. As long as your 
MemUsedPct
stays below 80-90% during idle times, this is healthy. StarRocks will automatically evict the Data Cache and trigger 
tcmalloc
to release memory if a large query suddenly needs it. Rocky's Advice: Don't try to lower this number by clearing caches. A "full" memory is a fast memory in StarRocks. Only worry if 
MemUsedPct
hits 95%+ and you start seeing "Memory Limit Exceeded" (MLE) errors in your query logs. References • administration/management/monitoring/metrics.md • administration/management/resource_management/Memory_management.md • sql-reference/System_variable.md
r
@Rocky we will need higher concurrency in the bakcend. we need to be able to run a bunch of jobs at once. is that a problem?
r
Running a large number of backend jobs at once is not a problem, provided you configure Query Queues. Without a queue, if you hit your 
concurrency_limit
, StarRocks will simply reject new queries with an error. With a queue, those "extra" jobs will simply wait their turn in a "Pending" state, which is exactly what you want for latency-tolerant backend work. Since you are on v4.0.8, you have access to the most advanced version of this feature. Here is how to set it up: 1. Enable the Global Queue Switch First, tell StarRocks to allow queuing instead of failing queries. Run these as a user with 
SYSTEM
level permissions:
Copy code
sql
-- Enable queuing for SELECT and INSERT (loading) jobs
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL enable_query_queue_load = true;
2. Update your Backend Resource Group You can now increase your 
concurrency_limit
. For your 3 nodes with 16 cores each, a limit of 20 to 30 is a safe starting point for backend jobs.
Copy code
sql
ALTER RESOURCE GROUP backend_rg WITH (
    "concurrency_limit" = "25" -- Allows 25 jobs to run at once across the cluster
);
Note: In v4.0.8, the Leader FE manages this collectively, so it won't let the cluster get overwhelmed even if jobs hit different nodes. 3. Use "Dynamic Parallelism" (The Secret Sauce) This is the best feature for high-concurrency environments. Instead of every query trying to use all 16 cores (which causes context switching lag), StarRocks can automatically lower the parallelism (
pipeline_dop
) of each query when the cluster is busy. Enable this by setting these variables:
Copy code
sql
-- When the number of active tasks is high, automatically 
-- reduce the number of cores each query uses.
SET GLOBAL query_queue_driver_low_water = 128; -- (8 * 16 cores)
SET GLOBAL query_queue_driver_high_water = 256; -- (16 * 16 cores)
4. Monitor the Queue If you start running "a bunch of jobs," you can see which ones are waiting using:
Copy code
sql
SELECT * FROM information_schema.resource_group_usage;
Look for the 
concurrency_overflow_count
. If this number is high but your 
CpuUsedPct
is low, you can safely increase your 
concurrency_limit
further. Important: Watch your Memory The only real "danger" of high concurrency in your specific setup is memory. • You have 91.8GB per node. • If you run 25 jobs at once, and each job decides to use 4GB, you will hit 100GB and the node will OOM (Out of Memory). • Rocky's Recommendation: Keep your 
big_query_mem_limit
strictly enforced in the backend resource group so that 10-20 concurrent jobs can't collectively exceed the 91.8GB limit. References • administration/management/resource_management/query_queues.md • administration/management/resource_management/resource_group.md • administration/management/resource_management/Memory_management.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md References • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md
r
@Rocky you had recommended 40GB big_query_mem_limit in the backend...are you changing that rec?
r
Yes, I am refining that recommendation. While 40GB was a good "rogue query" safety fence for 10 concurrent jobs, running 25 jobs at once changes the math. Here is why: The "Shared Budget" Problem You have 91.8GB total on the node. * Backend Group Budget: 60% of that is ~55GB. * The Conflict: If you keep 
big_query_mem_limit
at 40GB and run 25 jobs, a single heavy job could grab 40GB for itself. This leaves only 15GB total for the other 24 jobs (about 600MB each). * Result: The other 24 jobs will likely fail with "Memory Limit Exceeded" errors because the group hit its 55GB limit, even if those individual queries are small. Updated Recommendation for High Concurrency (25+ jobs) If you want high concurrency for backend jobs, you should move from a "fail-fast" strategy to a "spill-to-disk" strategy. 1. Lower the Per-Query Limit Lower 
big_query_mem_limit
to 20GB. This ensures no single job can consume more than ~35% of the backend's total memory budget on a node, leaving more room for the other 24 jobs. 2. Enable Intermediate Spilling (Critical for Concurrency) Since your backend jobs are latency-tolerant, you should enable spilling. Instead of a query failing when it runs out of memory, StarRocks will temporarily write the data to your 3.1TB local disk and continue. Update your Resource Group:
Copy code
sql
ALTER RESOURCE GROUP backend_rg WITH (
    "concurrency_limit" = "25",
    "big_query_mem_limit" = "21474836480", -- 20GB (Lowered for fairness)
    "spill_mem_limit_threshold" = "0.8"     -- Start spilling when group hits 80% of its 55GB
);
Set Session Variables for the Jobs: Ensure your backend job user/session has these enabled:
Copy code
sql
SET GLOBAL spill_mode = 'auto'; -- Enable automatic spilling to disk
The New Math for your 91.8GB Nodes: * 0GB - 55GB (Used by Backend Group): Performance is fast (all in memory). * At 44GB (80% of 55GB): Queries start spilling to your 3.1TB disk. They slow down, but they don't fail. * At 20GB (Single Query): If a single job is so massive it hits 20GB even with spilling, it is killed as a "rogue" to protect the system. Summary of the Change: * Old Rec (40GB): "I have few jobs; let them be as fast/big as possible, but kill them if they threaten the whole node." * New Rec (20GB + Spilling): "I have many jobs; keep them under a tighter leash so they share memory fairly, and use the 3.1TB disk as a safety valve so they finish without crashing." References • administration/management/resource_management/resource_group.md • administration/management/resource_management/spill_to_disk.md • faq/resource_isolation_faq.md • administration/management/resource_management/Memory_management.md -management) • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md References • administration/management/resource_management/resource_group.md • release_notes/release-4.0.md • faq/resource_isolation_faq.md
r
@Rocky ok please summarize your final recommendation, including settings i should change globally, plus the resource group setups
r
To handle high concurrency for backend jobs on your v4.0.8 cluster while managing your 91.8GB nodes effectively, here is the summarized recommendation. The strategy shifts from "kill heavy queries to save the node" to "throttle and spill queries to keep the cluster stable." 1. Global Settings (The Foundation) Run these commands to enable the safety valves (queuing and spilling) for all backend activity.
Copy code
sql
-- 1. Enable Query Queuing (Prevents overloading the CPU/Mem)
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL enable_query_queue_load = true;

-- 2. Enable Intermediate Spilling (Safety valve for memory)
SET GLOBAL enable_spill = true;
SET GLOBAL spill_mode = 'auto';

-- 3. Enable Dynamic Parallelism (Reduces per-query overhead when busy)
-- These use 8x and 16x your core count (16 cores * 3 nodes = 48 total cores)
SET GLOBAL query_queue_driver_low_water = 128;
SET GLOBAL query_queue_driver_high_water = 256;
2. Resource Group Configuration (
backend_rg
) This setup allows up to 25 jobs to run at once. It forces them to share the ~55GB memory pool (60% of 91.8GB) and starts spilling data to disk if they collectively use too much.
Copy code
sql
ALTER RESOURCE GROUP backend_rg WITH (
    -- Concurrency & Queueing
    "concurrency_limit" = "25",
    
    -- Memory Limits
    "mem_limit" = "60%", -- Total budget for this group on the node
    "big_query_mem_limit" = "21474836480", -- 20GB limit per individual query (protection)
    
    -- Spilling (The "Safety Valve")
    "spill_mem_limit_threshold" = "0.7" -- Start spilling if group uses > 70% of its 55GB
);
3. Critical Backend (BE) Check For spilling to work, you must ensure your nodes have a designated directory for temporary data. Check your 
be.conf
on each node: * Check: 
grep "spill_local_storage_dir" be.conf
* Recommendation: If it's not set, point it to your 3.1TB disk (e.g., 
spill_local_storage_dir=/data/starrocks/spill
). Note: Requires a BE restart to take effect. ────────── Why this is the "Final" recommendation: 1. Fairness: By lowering 
big_query_mem_limit
from 40GB to 20GB, you prevent one massive job from starving the other 24 jobs in your high-concurrency setup. 2. Stability: The 
concurrency_limit
of 25 ensures you don't hit the 91.8GB RAM limit by context-switching 100 queries at once. Extra jobs will simply wait in the queue. 3. No More "MLE" Errors: With 
enable_spill
and 
spill_mem_limit_threshold
set, StarRocks will use your 3.1TB local disk as "overflow RAM" rather than failing the query with a "Memory Limit Exceeded" (MLE) error. 4. Greedy but Managed: The 
MemUsedPct
will still look high (~52%) because of the Data Cache, but these settings ensure that the Query Pool (the part that actually does the work) is strictly regulated. How to monitor the result: Run this to see if jobs are queuing or how much memory they are using in real-time:
Copy code
sql
SELECT * FROM information_schema.resource_group_usage;
If you see 
concurrency_overflow_count
growing, it means your 25 limit is working—backend jobs are waiting their turn rather than crashing your nodes crashing. crashing the queue rather than crashing your cluster. References • administration/management/resource_management/resource_group.md • sql-reference/System_variable.md • administration/management/BE_configuration.md • administration/management/resource_management/spill_to_disk.md
r
@Rocky if i just have a single user, how do i ensure that that jobs traffic lands in the jobs group, and web traffic on web?
r
To route traffic from a single user into two different resource groups (web and jobs), you have three primary methods. I recommend Method 1 (Hints) for the cleanest separation without needing multiple database connections. Method 1: Using Query Hints (Recommended) You can leave the user's default resource group as 
web_rg
, and for specific "jobs" queries, simply add a hint. This is the most granular way to handle a single user. * How to run a job query:
Copy code
sql
    SELECT /_+ SET_VAR(resource_group = 'backend_rg') _/ *
    FROM your_table
    WHERE ...;
_StarRocks will ignore the classifier matching and force this query into the 
backend_rg
._ ────────── Method 2: Session-Level Routing If your backend job runner opens its own connection/session, you can set the group once at the start of the session. All subsequent queries in that connection will land in that group. * In the Job Runner code:
Copy code
sql
    SET resource_group = 'backend_rg';
    -- All following queries in this connection now use the jobs group
    INSERT INTO ... SELECT ...;
────────── Method 3: Automatic Routing via Classifiers If you don't want to change your code, you can use Classifiers to route based on where the query is coming from or what it is touching. 1. By Database: If jobs primarily target a specific database (e.g., 
analytics_db
) and web traffic targets another (e.g., 
app_db
).
Copy code
sql
    -- Update the backend group to catch all queries against the analytics DB
    ALTER RESOURCE GROUP backend_rg ADD (db='analytics_db');
2. By Source IP: If your web servers and your job runners have different IP addresses or CIDR blocks.
Copy code
sql
    -- Update the backend group to catch queries from the job runner server
    ALTER RESOURCE GROUP backend_rg ADD (source_ip='192.168.1.50/32');
Summary of Priority If you use multiple methods, StarRocks follows this priority: 1. Query Hint (
SET_VAR
) — Highest Priority 2. Session Variable (
SET resource_group
) 3. Classifier Match (IP, User, DB, etc.) 4. Default Group (
default_wg
) — Lowest Priority Rocky's Tip: Use Method 1 for ad-hoc jobs and Method 2 for scheduled batch processes that use a dedicated connection pool. This keeps your "web" traffic fast and responsive in its own lane! References • best_practices/query_tuning/query_hint.md • faq/resource_isolation_faq.md • sql-reference/System_variable.md • knowledge/usage/best-practice.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md
r
@Rocky when you say "default" rg what do you mean? how do i setup the non-default ones?
r
In StarRocks, resource groups allow you to carve up your cluster's CPU and memory so that one type of work (like a heavy ETL job) doesn't starve another (like a fast web dashboard). Here is the breakdown of what "default" means and how to set up your own. 1. What is the "Default" Resource Group? There is a system-defined group called 
default_wg
. * The Catch-all: Any query that does not match a specific rule (classifier) or doesn't have a hint attached to it automatically falls into this group. * Default Behavior: By default, 
default_wg
has no limits—it can technically use 100% of the node's resources. In a production environment, you usually want to "tame" it or ensure your important traffic is moved out of it. 2. How to Set Up "Non-Default" (Custom) Groups To create your own groups (like 
web_rg
and 
backend_rg
), you use the 
CREATE RESOURCE GROUP
command. Each group needs two things: Resource Limits and Classifiers. Step 1: Create the groups Here is how you would define your two groups:
Copy code
sql
-- 1. Create the Web Group (High priority/low latency)
CREATE RESOURCE GROUP web_rg
WITH (
    "cpu_weight" = "10",        -- Higher weight gets more CPU cycles
    "mem_limit" = "30%",        -- Reserve 30% of RAM for web queries
    "concurrency_limit" = "100" -- Allow many small queries
);

-- 2. Create the Backend Group (Lower priority/heavy lifting)
CREATE RESOURCE GROUP backend_rg
WITH (
    "cpu_weight" = "2",         -- Lower weight
    "mem_limit" = "60%",        -- Larger RAM pool for big joins
    "concurrency_limit" = "25",
    "big_query_mem_limit" = "21474836480" -- 20GB individual query cap
);
Step 2: Set up Classifiers (The Routing Rules) Classifiers tell StarRocks which query belongs where. Since you have a single user, you can't route by "user name," so you use other identifiers. Option A: Route by Database name If your web app queries the 
prod_tables
DB and your jobs query `staging_tables`:
Copy code
sql
ALTER RESOURCE GROUP web_rg ADD (db='prod_tables');
ALTER RESOURCE GROUP backend_rg ADD (db='staging_tables');
Option B: Route by IP Address If your web server is at 
10.0.0.5
and your job runner is at `10.0.0.50`:
Copy code
sql
ALTER RESOURCE GROUP web_rg ADD (source_ip='10.0.0.5/32');
ALTER RESOURCE GROUP backend_rg ADD (source_ip='10.0.0.50/32');
Option C: The "Default User" fallback You can also assign a specific group to a user so they land there by default:
Copy code
sql
-- This makes 'my_user' land in web_rg if no other rules match
ALTER RESOURCE GROUP web_rg ADD (user='my_user');
3. How to check your setup Once you've run these, you can verify that your rules are active and see where queries are actually landing:
Copy code
sql
-- Check your group configurations and weights
SHOW RESOURCE GROUPS;

-- Check where current queries are running
SELECT query_id, resource_group, state, mem_usage 
FROM information_schema.resource_group_usage;
Summary of the Flow 1. A query arrives from your user. 2. StarRocks looks for a hint (e.g., 
/_+ SET_VAR(resource_group='backend_rg') _/
). If found, it goes there. 3. StarRocks checks Classifiers: Does the IP or DB match 
web_rg
or 
backend_rg
? 4. The Fallback: If nothing matches, the query lands in 
default_wg
. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md
r
@Rocky i have an existing group with
Copy code
classifiers: (id=125197, weight=2.1, user=test, query_type in (INSERT))
where is the "weight" classifier coming from?
r
The "weight" you see in the classifier string is an internal priority score (also called "degree of matching") that StarRocks calculates for every classifier. When a query is submitted, StarRocks checks it against all classifiers. If a query matches multiple classifiers, it chooses the one with the highest weight. This ensures that "more specific" rules (like a rule for a specific database) take precedence over "broad" rules (like a rule for a whole user). How the weight 
2.1
is calculated In your specific case: 
(user=test, query_type in (INSERT))
has a weight of 2.1. This is calculated using the following internal point system: 1. User Match (
user=test
): +1.0 point 2. Query Type Match (
query_type in (INSERT)
): +1.1 points * The formula for query type is: 
1.0 + (0.1 / Number of types in the classifier)
. * Since you only have one type (
INSERT
), it is 
1.0 + (0.1 / 1) = 1.1
. _ _Note: If you had 
query_type in (INSERT, SELECT)
, the weight would drop to 
2.05
because the rule is less specific.* 3. Total: 
1.0 + 1.1 = 2.1
Other Common Weight Components If you add more conditions to your classifier, the weight increases, making that rule "stronger": Condition | Points Added to Weight | Notes Database (
db
) | +10.0 | Designed to be the strongest override. User (
user
) | +1.0 | Matches the specific username. Role (
role
) | +1.0 | Matches the user's role. IP Address (
source_ip
) | 1.0 to 1.5 | More specific CIDR masks (like 
/32
) get higher scores than broad ones (like 
/8
). Cost Range | +1.0 | Matches estimated CPU/Mem cost. Why this matters If you ever create a second classifier that also matches the 
test
user, the weights determine which group "wins." * Example: If you added a new classifier for 
db='sales'
, its weight would be 10.0 (or higher if combined with user). Even if the user is 
test
, the 
db='sales'
rule would win because its weight is much higher than 
2.1
. You can view these weights and how they resolve for your queries by running:
Copy code
sql
-- See the weight for all active classifiers
SHOW RESOURCE GROUPS;

-- See which resource group (and weight) a specific query used
EXPLAIN VERBOSE <your_query>;
References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md • sql-reference/sql-functions/string-functions/ngram_search.md • table_design/indexes/vector_index.md • sql-reference/sql-functions/like-predicate-functions/like.md • sql-reference/sql-functions/math-functions/cos_similarity.md
r
@Rocky weird, im not using that user though, why would that group be useD?
Copy code
mysql> show user;
+------------+
| User       |
+------------+
| 'root'@'%' |
+------------+
1 row in set (0.01 sec)
r
If you are logged in as 
root
, that specific classifier (
user=test, query_type in (INSERT)
) will not be used for your queries. In StarRocks, a classifier only matches if all of its conditions are met. Since your current user is 
root
and the classifier requires 
user=test
, it's a mismatch. If you are seeing your queries land in that group anyway, it is likely due to one of the following reasons: 1. There is another classifier in that group A resource group can have multiple classifiers. Run 
SHOW RESOURCE GROUPS ALL;
to see all rules for that group. You might find another rule like: * 
(db='your_db')
— Matches any user (including 
root
) querying that database. * 
(query_type in (INSERT))
— Matches any user running an 
INSERT
if no 
user
is specified in that specific rule. * 
(source_ip='...')
— Matches based on where you are connecting from. 2. A Session Variable is set If you (or your application/IDE) ran the following command earlier in the session, StarRocks ignores the classifiers and forces the traffic into that group:
Copy code
sql
SET resource_group = 'your_group_name';
This stays active until the connection is closed or changed. 3. A Query Hint is being used If your SQL contains a hint, it overrides the user-based classifier:
Copy code
sql
SELECT /*+ SET_VAR(resource_group='your_group_name') */ ...
4. How to confirm where your query is actually landing Don't rely on 
SHOW RESOURCE GROUPS
to see what is currently happening (that just shows the "rules"). To see which group your 
root
user is actually being assigned to right now, run:
Copy code
sql
-- 1. Check your current session's group
SELECT @@resource_group;

-- 2. Check the plan for a specific query (Look for 'resource_group' in the output)
EXPLAIN VERBOSE SELECT * FROM your_table;

-- 3. Check historical usage
SELECT query_id, user, resource_group, query_type 
FROM information_schema.resource_group_usage 
WHERE user = 'root';
Why the weight is 2.1: As mentioned before, that 
2.1
is just the "strength" score of that rule's definition. StarRocks pre-calculates this so that if a user named 
test
runs an 
INSERT
, it knows this rule is more specific (stronger) than a rule that just says 
user=test
. Since you aren't 
test
, the rule simply doesn't apply to you. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md
r
@Rocky if the user is associated with 3 different groups with no other classifiers, how is one chosen?
r
If a user is associated with three different resource groups and the classifiers are identical (e.g., they all just specify 
user='test'
), StarRocks handles the tie as follows: 1. The Tie-Breaker: Random Selection When multiple classifiers match a query with the exact same weight (degree of matching), StarRocks randomly selects one of the matching resource groups. In your case: * Group A Classifier: 
(user='test')
→ Weight: 1.0 * Group B Classifier: 
(user='test')
→ Weight: 1.0 * Group C Classifier: 
(user='test')
→ Weight: 1.0 Because all three have a weight of 1.0, the query could land in any of them. There is no "first-created wins" or alphabetical priority. 2. How to "Force" a Specific Group If you want to stop the random behavior and ensure your user lands in a specific group, you have three options: Option A: Increase the weight of one group Add a more specific condition to the group you want to prioritize. For example, specify the database or query type:
Copy code
sql
-- This group now has a weight of 11.0 (10 for DB + 1 for user)
-- It will always beat the 1.0 weight groups when querying 'prod_db'
ALTER RESOURCE GROUP Group_A ADD (db='prod_db');
Option B: Set a session variable You can force your current session into a specific group. This overrides all classifiers:
Copy code
sql
SET resource_group = 'Group_B';
Option C: Use a Query Hint Force a single specific query into a group:
Copy code
sql
SELECT /*+ SET_VAR(resource_group='Group_C') */ * FROM table;
3. How to See Which One Won If you want to see which group the system actually picked for a query, you can use `EXPLAIN VERBOSE`:
Copy code
sql
EXPLAIN VERBOSE SELECT * FROM my_table;
Look for the 
ResourceGroup
field in the output. It will show you exactly which group the query was assigned to and why. Summary of Weighting (Recap) StarRocks uses these points to calculate the weight. If these are identical across your groups, you get the random selection: * Database (
db
): +10.0 * User (
user
): +1.0 * Role (
role
): +1.0 * Query Type: ~1.1 * Plan Cost: +1.0 References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md
r
@Rocky i have big_query_cpu_second_limit set to 10 minutes and have seem some queries time out after shorter than that. why would that be? does the number of backend matter here?
r
The reason your queries are timing out "early" is that 
big_query_cpu_second_limit
does not measure wall-clock time (the time on your watch); it measures cumulative CPU usage across all cores on a single machine. Here is why your query is hitting the limit faster than 10 minutes: 1. CPU Time vs. Wall-Clock Time StarRocks queries are highly parallel. If your query uses 10 CPU cores on a single Backend (BE) node, it consumes 10 CPU seconds for every 1 second of real time. * Example: With a 600-second limit (10 minutes), a query using 20 cores will be terminated after only 30 seconds of real time ($20 \times 30 = 600$). * The more cores your BE has and the more parallel your query is, the faster you will hit this limit. 2. It is a "Per-BE" Limit This limit is applied per Backend node, not globally across the cluster. * The number of backends does matter for your overall cluster capacity, but it doesn't "spread out" this specific limit. * If your query is skewed (e.g., most of the processing happens on one node due to a specific join key or partition), that one node will hit the 600s CPU limit and kill the query, even if the other nodes have used 0 CPU seconds. 3. Check for 
query_timeout
 You might also be hitting the standard 
query_timeout
. * By default, 
query_timeout
is often set to 300 seconds (5 minutes) in many environments. * 
query_timeout
measures wall-clock time. If your query takes longer than this, it will be killed regardless of how much CPU it used. * To check your current timeout: 
SHOW VARIABLES LIKE 'query_timeout';
How to troubleshoot To see exactly why the query was killed, check the 
ErrorCode
or 
ErrorMsg
in the FE audit log (
fe.audit.log
). * If it says 
Big query exceeded cpu_second_limit
, then the parallel CPU accumulation explained above is the culprit. * If it says 
Query exceeded timeout
, you need to increase the 
query_timeout
session variable. Recommendation If you want a query to be allowed to run for 10 minutes of "real time" regardless of how many cores it uses, you should: 1. Increase 
query_timeout
to 
600
. 2. Set 
big_query_cpu_second_limit
to a much higher value (e.g., 
600 * number_of_cores_per_be
) or set it to 
0
to disable that specific resource group to disable it. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • administration/management/monitoring/metrics.md References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md • administration/management/monitoring/metrics.md
r
@Rocky what resource group does the auto analyzer use? what can i tweak if analyze querys are hitting
Copy code
com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Statistics query fail | Error Message [[reason=INTERNAL_ERROR] [msg=exceed big query cpu limit:
r
The auto analyzer (background statistics collection) in StarRocks typically runs under the 
root
 user identity and, by default, is assigned to the 
default_wg
 resource group. When you see the error 
exceed big query cpu limit
, it means the internal statistics query (which scans your tables to count rows, find min/max, etc.) has consumed more cumulative CPU seconds on a single BE than the group's 
big_query_cpu_second_limit
allows. How to identify the culprit To confirm exactly which user and group the analyzer is using, check your FE audit log (
fe/log/fe.audit.log
or query the 
starrocks_audit_tbl__
if you use AuditLoader). Look for queries like 
SELECT ... FROM _statistics_...
or 
ANALYZE TABLE ...
. * User: Likely 
'root'@'%'
* ResourceGroup: Likely 
default_wg
What you can tweak 1. Increase the limit for the resource group If the analyzer is hitting the limit in 
default_wg
, you can increase it. Note that 
0
means unlimited.
Copy code
sql
ALTER RESOURCE GROUP default_wg WITH (
    'big_query_cpu_second_limit' = '3600' -- Increase to 1 hour, or 0 for unlimited
);
2. Create a dedicated Resource Group for Statistics If you want to keep strict limits on other 
root
queries but allow the analyzer more room, create a specific group for the 
root
user:
Copy code
sql
CREATE RESOURCE GROUP stats_wg
TO (user='root')
WITH (
    'cpu_weight' = '10',
    'mem_limit' = '20%',
    'big_query_cpu_second_limit' = '0' -- No limit for system tasks
);
_Note: If you have other users running as 
root
, they will also move to this group. You can differentiate by 
query_type
if needed._ 3. Adjust Statistics Collection Parallelism If the collection is too aggressive, you can slow it down so it hits the limit less frequently (or speed it up if you have the head room):
Copy code
sql
-- Adjust parallelism (default is often 1)
SET GLOBAL statistic_collect_parallel = 1;
4. Switch to Sampled Collection If full statistics collection on a massive table is what's hitting the CPU limit, tell StarRocks to use sampling instead. Sampled collection is much lighter on CPU.
Copy code
sql
-- Change the default behavior to sampled for large tables
ADMIN SET FRONTEND CONFIG ("enable_collect_full_statistic" = "false");
You can also manually trigger a sampled analyze for a specific problematic table:
Copy code
sql
ANALYZE SAMPLE TABLE your_table_name;
5. Exclude problematic tables If certain tables are too large and not critical for CBO right now, you can exclude them from auto-collection using a regex pattern in the FE config 
statistic_exclude_pattern
. References • administration/management/resource_management/resource_group.md • using_starrocks/Cost_based_optimizer.md • administration/management/FE_configuration.md References • administration/management/logs.md • administration/management/audit_loader.md • faq/operation_maintenance_faq.md
r
@Rocky is it possible the analyze tasks are stealing resources from my main app? whats the general recommendation for resources for those tasks
r
Yes, it is absolutely possible for auto-analyze tasks to "steal" resources from your main application. Because these tasks run as background queries (typically under the 
root
user), they compete for the same CPU and memory as your application queries if they aren't isolated. Why they "steal" resources 1. Shared Resource Group: By default, analyze tasks often fall into the 
default_wg
or are matched by the same classifiers as other 
root
activities. 2. Resource Contention: If a massive table is being analyzed, StarRocks will scan the data to compute statistics. This consumes CPU cycles and disk I/O that your app needs. 3. Parallelism: The variable 
statistic_collect_parallel
(default 1) controls how many analyze tasks run at once. If increased, it can significantly spike CPU usage. ────────── General Recommendations & Tweaks To prevent analyze tasks from impacting your app while solving the 
big query cpu limit
error, follow these best practices: 1. Isolate the Analyzer in a "Low Priority" Group The best way to stop "stealing" is to put the analyzer in its own resource group with a low 
cpu_weight
but a high (or unlimited) 
big_query_cpu_second_limit
.
Copy code
sql
-- 1. Create a group with low weight so it only uses "leftover" CPU
CREATE RESOURCE GROUP stats_wg
TO (user='root') -- Assuming analyzer runs as root
WITH (
    'cpu_weight' = '1',              -- Give it very low priority
    'mem_limit' = '10%',             -- Limit memory to prevent OOM
    'big_query_cpu_second_limit' = '0' -- 0 = Unlimited (fixes your error)
);
2. Schedule During Off-Peak Hours You can restrict the auto-analyzer to only run when your app is less busy (e.g., at night).
Copy code
sql
-- Run only between 1 AM and 5 AM
ADMIN SET FRONTEND CONFIG ("statistic_auto_analyze_start_time" = "01:00:00");
ADMIN SET FRONTEND CONFIG ("statistic_auto_analyze_end_time" = "05:00:00");
3. Lower the Collection Parallelism If you see too many BE cores being used by stats, ensure your parallelism is set to 1.
Copy code
sql
SET GLOBAL statistic_collect_parallel = 1;
4. Force "Sampled" instead of "Full" Stats For very large tables, a "Full" analyze is extremely expensive. You can lower the threshold at which StarRocks switches from Full to Sampled collection.
Copy code
sql
-- Default is often 100GB. Lowering this forces sampling on smaller tables.
ADMIN SET FRONTEND CONFIG ("statistic_max_full_collect_data_size" = "53687091200"); -- 50GB
5. Monitor via Audit Logs To confirm if the analyzer is the one hitting the limit, search your 
fe.audit.log
for queries hitting the 
_statistics_
database. You will see the 
ResourceGroup
and 
CPUTime
there, which will confirm if your new 
stats_wg
is working correctly. References • using_starrocks/Cost_based_optimizer.md • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • using_starrocks/Cost_based_optimizer.md • administration/management/FE_configuration.md
r
@Rocky i added that resource group, how can i tell if its working as we want?
r
To verify if your new resource group is working correctly for the analyzer, you can use a combination of live monitoring, historical logs, and metrics. 1. Check Live Resource Usage (v3.1.4+) The most direct way to see if your 
stats_wg
is active is to run:
Copy code
sql
SHOW USAGE RESOURCE GROUPS;
* What to look for: Look for 
stats_wg
in the 
Name
column. * Success: If you see positive numbers for 
BEInUseCpuCores
or 
BERunningQueries
under 
stats_wg
while an analyze task is running, the classifier is working. 2. Check the Audit Log (Historical) The FE audit log (
fe.audit.log
) records the resource group for every query. Search for queries targeting the statistics database:
Copy code
bash
grep "_statistics_" fe.audit.log | grep "stats_wg"
* What to look for: In the 
ResourceGroup
column of the log, you should see 
stats_wg
. * Note: If you have the 
AuditLoader
extension enabled, you can query this directly:
Copy code
sql
    SELECT queryId, resourceGroup, querySql
    FROM starrocks_audit_tbl__
    WHERE querySql LIKE '%_statistics_%'
    ORDER BY startTime DESC LIMIT 10;
3. Use 
EXPLAIN
to Test Classifiers You can "dry run" a query as the 
root
user to see which group StarRocks would assign it to:
Copy code
sql
EXPLAIN VERBOSE SELECT * FROM _statistics_.table_statistic_v1;
* What to look for: At the very top of the output, look for a line like: 
Resource Group: stats_wg
4. Monitor "Big Query" Triggers Since you created this group to avoid the "big query cpu limit" error, you should check if queries are still being flagged as "big queries" (even if they aren't being killed now). * Metric: 
resource_group_bigquery_count
* Where to find it: In your Prometheus/Grafana dashboard or by checking the BE metrics endpoint (
http://<be_ip>:<be_http_port>/metrics
). * Interpretation: If this count stops increasing for your main application's group and only moves for 
stats_wg
, you have successfully isolated the impact. 5. Check Active Queries While an analyze task is running, check the processlist:
Copy code
sql
SHOW FULL PROCESSLIST;
In the 
ResourceGroup
column, the internal stats queries should now show 
stats_wg
instead of 
default_wg
. What if it’s NOT working? If you still see the analyzer in `default_wg`: * Classifier Weight: Ensure the 
stats_wg
classifier has a higher "match degree" than other groups. A classifier with 
user='root'
is quite specific, but if another group has 
db='_statistics_'
, that would have a higher priority (weight 10). * Pipeline Engine: Ensure the Pipeline Engine is enabled (
SET GLOBAL enable_pipeline_engine = true;
), as resource groups require it. References • administration/management/resource_management/resource_group.md • administration/management/monitoring/metrics.md References • administration/management/resource_management/resource_group.md • administration/management/monitoring/metrics.md • administration/management/audit_loader.md References • administration/management/resource_management/resource_group.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md
r
@Rocky
Copy code
SELECT queryId, resourceGroup, querySql
    FROM starrocks_audit_tbl__
    WHERE querySql LIKE '%_statistics_%'
    ORDER BY startTime DESC LIMIT 10;
doesnt work and i dont see the runing analyze task in SHOW FULL PROCESSLIST; even though its at show analyze status;
r
The reason you aren't seeing them in 
SHOW FULL PROCESSLIST
is that this command primarily lists active client connections. Background tasks like the auto-analyzer don't always hold a standard client session. To see the resource group mapping for internal tasks and verify if your new group is being used, use these methods instead: 1. The "Current Queries" View This is the most reliable way to see what's actually running on the engine, including internal SQL triggered by the analyzer.
Copy code
sql
SHOW PROC '/current_queries';
-- or for a cluster-wide view
SHOW PROC '/global_current_queries';
* What to look for: Look for queries where the 
Database
is 
_statistics_
or the 
User
is 
root
. * The Check: Look at the 
ResourceGroup
 column. It should show 
stats_wg
. 2. Check the physical Audit Log If 
starrocks_audit_tbl__
doesn't work, it means you haven't enabled the Audit Loader extension. You can still check the raw logs on your FE server: 1. Log into your FE leader node. 2. Go to the 
fe/log/
directory. 3. Run this command to see which group the statistics queries are hitting:
Copy code
bash
    grep "_statistics_" fe.audit.log | awk -F'|' '{print "Time:" $1 " | User:" $3 " | Group:" $14 " | SQL:" $16}' | tail -n 20
(Note: The column index might vary slightly depending on your version, but look for the field containing your resource group name.) 3. Verify via "Show Usage" Even if you can't catch the query in the act, StarRocks tracks cumulative usage per group:
Copy code
sql
SHOW USAGE RESOURCE GROUPS;
If you see 
BEInUseCpuCores
 or 
MemUsageBytes
 fluctuating for 
stats_wg
while 
SHOW ANALYZE STATUS
says "RUNNING", then the group is correctly capturing the workload. Why 
SHOW ANALYZE STATUS
is different 
SHOW ANALYZE STATUS
shows the metadata task (the management of the analyze job). The actual heavy lifting is done by internal SQL queries (e.g., 
SELECT ... FROM table
) that the task submits to the engine. It is those internal SQL queries that get assigned to a Resource Group, not the "task" itself. If it’s still hitting 
default_wg
 If the logs show the analyzer is still in 
default_wg
, your classifier might need to be more specific. Try adding the database to the classifier:
Copy code
sql
ALTER RESOURCE GROUP stats_wg ADD (user='root', db='_statistics_');
References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • using_starrocks/Cost_based_optimizer.md References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/logs.md References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md • sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md
r
@Rocky i think they are usign the default read RG
Copy code
mysql> SHOW USAGE RESOURCE GROUPS;
+-----------------+-------+--------------------------------------------------------------------------------------------------------+-----------------+-----------------+------------------+
| Name            | Id    | Backend                                                                                                | BEInUseCpuCores | BEInUseMemBytes | BERunningQueries |
+-----------------+-------+--------------------------------------------------------------------------------------------------------+-----------------+-----------------+------------------+
| default_read_rg | 50320 | - | 3.608           | 282422256       | 1                |
| default_read_rg | 50320 | - | 7.832           | 513481328       | 2                |
| default_read_rg | 50320 | - | 3.772           | 283965744       | 2                |
+-----------------+-------+--------------------------------------------------------------------------------------------------------+-----------------+-----------------+------------------+
3 rows in set (0.04 sec)

mysql> show resource groups all;
+------------------+---------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+------------------------------------------------+
| name             | id      | cpu_weight | exclusive_cpu_cores | mem_limit | big_query_cpu_second_limit | big_query_scan_rows_limit | big_query_mem_limit | concurrency_limit | spill_mem_limit_threshold | classifiers                                    |
+------------------+---------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+------------------------------------------------+
| default_mv_wg    | 3       | 1          | 0                   | 80.0%     | 0                          | 0                         | 0                   | null              | 80%                       | (id=0, weight=0.0)                             |
| default_read_rg  | 50320   | 8          | 0                   | 90.0%     | 3600                       | 0                         | 15000000000         | null              | 80%                       | (id=50321, weight=1.1, query_type in (SELECT)) |
| default_wg       | 2       | 16         | 0                   | 100.0%    | 0                          | 0                         | 0                   | null              | 100%                      | (id=0, weight=0.0)                             |
| default_write_rg | 50322   | 8          | 0                   | 90.0%     | 0                          | 0                         | 0                   | null              | 80%                       | (id=50323, weight=1.1, query_type in (INSERT)) |
| muncher_jobs_rg  | 1539138 | 10         | 0                   | 60.0%     | 3600                       | 1000000000                | 21474836480         | 10                | 70%                       | (id=1539139, weight=1.0, user=muncher)         |
| muncher_web_rg   | 1539136 | 0          | 4                   | 25.0%     | 120                        | 0                         | 0                   | 100               | 80%                       | (id=1539137, weight=1.0, user=muncher)         |
| stats_wg         | 1547836 | 1          | 0                   | 10.0%     | 0                          | 0                         | 0                   | null              | 100%                      | (id=1547837, weight=1.0, user=root)            |
+------------------+---------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+------------------------------------------------+
7 rows in set (0.04 sec)
r
You are correct. Based on your 
SHOW RESOURCE GROUPS ALL
output, the 
default_read_rg
is "winning" the classifier match because it has a higher weight (1.1) than your 
stats_wg
(1.0). Why this is happening * 
stats_wg
weight (1.0): You only specified 
user='root'
. * 
default_read_rg
weight (1.1): It specifies 
query_type in (SELECT)
. In StarRocks, a 
query_type
match is often weighted slightly higher (or adds to the base weight) to ensure specific query types are caught. * The Conflict: Since the analyze tasks run 
SELECT
queries as the 
root
user, they match both groups. Because 
1.1 > 1.0
, StarRocks picks 
default_read_rg
. The Fix: Make the 
stats_wg
classifier more specific To force the analyze tasks into 
stats_wg
, you need to add a classifier with more conditions. StarRocks always prefers the classifier with the highest number of matching conditions. Run this command to add a more specific classifier to your group:
Copy code
sql
-- This adds a classifier with 3 conditions. 
-- It will beat default_read_rg (1 condition) and your old root classifier (1 condition).
ALTER RESOURCE GROUP stats_wg ADD (user='root', db='_statistics_', query_type in ('SELECT'));
How to verify it worked: 1. Wait for an analyze task to run (or trigger one manually with 
ANALYZE TABLE <name>
). 2. Check usage again:
Copy code
sql
    SHOW USAGE RESOURCE GROUPS;
You should now see 
stats_wg
appearing in the 
Name
column with active CPU/Memory usage. 3. Clean up (Optional): Once you confirm it's working, you can remove the old, weaker classifier from 
stats_wg
to keep things tidy:
Copy code
sql
    -- Use the ID from your 'show resource groups all' output (1547837)
    ALTER RESOURCE GROUP stats_wg DROP (1547837);
Why this is better for your cluster: 1. Lower Priority: Your 
stats_wg
has 
cpu_weight = 1
, while 
default_read_rg
has 
8
. Moving the analyzer here means it will automatically "yield" CPU to your main app whenever the cluster is busy. 2. No Timeouts: Since 
stats_wg
has 
big_query_cpu_second_limit = 0
, your statistics collection won't be killed if it takes longer than the 3600s (1 hour) limit set on your 
default_read_rg
. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md • faq/resource_isolation_faq.md References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md
r
@Rocky will a running analyze be updated, or this will just kick in for the next that runs?
r
It depends on whether you mean the overall analyze task or the individual SQL queries it triggers. 1. The Short Answer * Individual Queries already on the BE: Any SQL query that is already "RUNNING" on the Backend (BE) will not change its resource group mid-execution. Classification happens at the very start of a query's lifecycle. * The Analyze "Task": An 
ANALYZE
operation is typically broken down into multiple internal SQL queries (e.g., scanning one partition at a time or different columns). Any sub-queries that have not yet started will pick up your new classifier and run in the 
stats_wg
. * Future Tasks: All future analyze runs will fully use the new group. 2. How "ALTER" works in StarRocks StarRocks distinguishes between changing limits and changing classifiers: * Updating Limits (
WITH
clause): If you change the 
cpu_weight
or 
mem_limit
of a group, that change usually takes effect immediately for all queries already running in that group. The BE scheduler dynamically adjusts its CPU sharing based on the new weights. * Updating Classifiers (
ADD/DROP
): This only affects new queries. When a query arrives at the FE, it is "stamped" with a resource group ID. It keeps that ID until it finishes. 3. How to check for the "switch" Since an analyze task often takes a while and runs multiple queries, you can actually watch the transition happen: 1. Run 
SHOW PROC '/current_queries';
(or 
SHOW PROC '/global_current_queries';
on newer versions). 2. In the 
ResourceGroup
column, you might see some internal root queries still in 
default_read_rg
(the ones that started first) and newer ones appearing in 
stats_wg
. 3. Once the old queries finish, 
SHOW USAGE RESOURCE GROUPS;
will show 
default_read_rg
dropping to 0 and 
stats_wg
taking over the load. Recommendation If the current analyze task is causing a performance issue and you want it to move to the new group right now, the fastest way is to: 1. Kill the task: 
KILL ANALYZE <job_id>;
(find the ID from 
SHOW ANALYZE STATUS
). 2. Restart it: Manually trigger it again with 
ANALYZE TABLE <table_name>;
. 3. Because it is a "new" start, all its internal queries will immediately hit the new classifier and be assigned to 
stats_wg
. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md • using_starrocks/Cost_based_optimizer.md References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md • using_starrocks/Cost_based_optimizer.md References • administration/management/resource_management/resource_group.md
r
@Rocky 
SHOW PROC '/global_current_queries';
isnt working (shared-data mode, v4.0.9)
r
If 
SHOW PROC '/global_current_queries';
is not returning results or is returning an error in v4.0.9 shared-data mode, it is usually due to one of the following three reasons: 1. The Audit Table requires an Extension The table 
starrocks_audit_tbl__
is not a built-in system table. It is part of the Audit Loader extension. * If you haven't manually installed the Audit Loader and created that table in a database (usually 
starrocks_monitor
), your 
SELECT
query will fail with 
Table ... doesn't exist
. * Verification: Run 
SHOW TABLES FROM starrocks_monitor;
(or wherever you intended to install it) to see if it exists. 2. Privileges and FE Leader * Permissions: You must have the 
ADMIN
or 
SYSTEM
role to view queries from other users or global cluster info. If you are logged in as a restricted user, these paths may appear empty or return "Access Denied." * Leader Node: While 
global_current_queries
is designed to aggregate data, try connecting directly to the FE Leader node and run 
SHOW PROC '/current_queries';
. Sometimes follower FEs have transient issues gathering global state in very high-load clusters. 3. Why Analyze might be "Invisible" 
SHOW ANALYZE STATUS
tracks the lifecycle of the statistics job, which can stay in "RUNNING" for a long time. However, the internal SQL queries it sends to the engine (the ones that would show up in 
current_queries
) are often: * Short-lived: For small tables or specific column types, the SQL finishes in milliseconds, making it very hard to "catch" in a manual 
SHOW PROC
command. * Metadata-only: Some statistics updates only touch the FE metadata and don't trigger a BE scan at all. These won't appear as "running queries" because they don't have an execution plan on the Backends. How to verify the Resource Group in v4.0.9 without those commands Since you can't see the running query, use the cumulative metrics to prove the 
stats_wg
is being used: 1. Check current usage snapshots: Instead of queries, look at the BE-side resource usage.
Copy code
sql
    SHOW USAGE RESOURCE GROUPS;
If you see 
BEInUseCpuCores
or 
BERunningQueries
increase for 
stats_wg
when you manually trigger 
ANALYZE TABLE <your_table>
, then the classifier is working. 2. Check the FE Audit Log (The "Ground Truth"): This is the only place that records every single internal query, even if it lasts only 1ms. * Find your FE 
log/
directory. * Look at 
fe.audit.log
. * Search for the database 
_statistics_
. You will see the 
ResourceGroup
column in the log entries.
Copy code
bash
    grep "_statistics_" fe.audit.log | tail -n 10
3. Check Query Queue (if enabled): If you have query queues enabled, you can use:
Copy code
sql
    SHOW RUNNING QUERIES;
This view often shows the 
ResourceGroupId
more reliably than the processlist. Summary Checklist * Is Audit Loader installed? If not, 
starrocks_audit_tbl__
won't work. * Is the classifier weighted correctly? Ensure you added the more specific classifier (
db='_statistics_'
) as mentioned in the previous step to beat the 
default_read_rg
weight. * Are you the root user? Ensure you have full permissions to see the 
proc
paths. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/resource_management/resource_group.md • administration/management/audit_loader.md • using_starrocks/running_queries.md References
r
@Rocky still using default_read_rg -- possible its not running root? or does the db='_statistics_' filter stop it from being used since the stats themselves need to look at tables outside of that db?
r
You've identified exactly why the previous fix didn't work. Your intuition is correct: the 
db='_statistics_'
filter is likely the reason the classifier isn't matching the expensive part of the 
ANALYZE
process. Why 
db='_statistics_'
is the problem When you run 
ANALYZE
, StarRocks does two things: 1. The Scan (Heavy): It runs internal 
SELECT
queries to scan your data. The 
db
context for these queries is the database where your table lives (e.g., 
sales_db
), not 
_statistics_
. 2. The Write (Light): Once it has the results, it `INSERT`s them into the 
_statistics_
database. Because the expensive table scan uses your actual database name, the classifier 
(user='root', db='_statistics_', query_type='SELECT')
fails to match the scan queries. It falls back to the next best match, which is 
default_read_rg
(because it matches the 
SELECT
type and has a higher weight of 1.1 than your base 
root
user classifier at 1.0). The Solution: "Beating" the weight To force `root`'s 
SELECT
queries (which include analyze scans) into 
stats_wg
, you need a classifier that matches the scan queries and has more conditions than the one in 
default_read_rg
. Run this command:
Copy code
sql
-- This classifier has 2 conditions (user + type).
-- This beats default_read_rg's 1 condition (type) and your old root classifier (1 condition).
ALTER RESOURCE GROUP stats_wg ADD (user='root', query_type in ('SELECT'));
How to confirm the identity of Analyze queries If it still hits the default group, it’s possible the internal tasks aren't running as 
root
. To be 100% sure in v4.0.9: 1. Check the FE Audit Log: This is the "source of truth." Look for the internal scan queries in your FE's 
log/fe.audit.log
.
Copy code
bash
    # Search for "SELECT" queries and see what user and ResourceGroup are listed
    grep "SELECT" fe.audit.log | tail -n 20
Look at the 
User
column and the 
ResourceGroup
column in the log. If you see a user other than 
root
(like a system internal user), change your classifier to match that user. 2. Verify via `EXPLAIN`: You can see which resource group a manual analyze will use by running:
Copy code
sql
    -- Replace with your table name
    EXPLAIN VERBOSE SELECT * FROM your_db.your_table;
(Note: 
EXPLAIN
on the 
ANALYZE
command itself isn't supported, but since the analyze uses a standard 
SELECT
scan, checking a 
SELECT
as 
root
will tell you where those internal queries are going). Summary of Rules * Database: Only use 
db
in a classifier if you want to isolate queries hitting a specific database. For 
ANALYZE
, the database context is the table's DB. * Weight: Your 
default_read_rg
has a weight of 1.1. Any classifier you want to "win" must have a match score higher than 1.1. A classifier with two matching conditions (like 
user
AND 
query_type
) will always beat a classifier with only one. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • using_starrocks/Cost_based_optimizer.md References • using_starrocks/Cost_based_optimizer.md • administration/management/resource_management/resource_group.md
r
@Rocky does the mem_limit in a resource groups apply across queries in the group, or its per query (so if i have mem_limit=10% does that mean each query gets a separate 10% or if 3 queries are running, together they can consume 10%)?
r
The 
mem_limit
in a resource group is group-wide (aggregated across all queries in that group on a single BE node), not per-query. Here is how it works for your example (
mem_limit=10%
): * Total Capacity: If 3 queries are running simultaneously in that group, their combined memory usage on that BE cannot exceed 10% of the BE's total query memory pool. * Enforcement: If Query A and Query B are already using 9% of the memory, and Query C tries to use more than the remaining 1%, the system will either trigger spilling to disk (if configured) or return a "Memory limit exceeded" error for one of the queries. * Per-Query Control: If you want to limit how much an individual query can take within that 10% pool, you can use the parameter 
big_query_mem_limit
within the same resource group. This acts as a "ceiling" for any single query. Summary Parameter | Scope | Behavior 
mem_limit
 | Group-wide | If 3 queries run, they share the 10%. 
big_query_mem_limit
 | Per-query | Limits a single query's max usage (e.g., set to 2GB). Note on Shared Data (v4.0+): In version 4.0, you may also see a 
mem_pool
parameter. This allows multiple resource groups to share the same memory "bucket" if they are assigned to the same pool, same pool name, further aggregating that aggregating the limit across groups. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md
Open in Slack
PreviousNext
Powered by