http://starrocks.io logo
Join Slack
Powered by
I want to understand resource groups queues better...
# questions-and-troubleshooting
r
I want to understand resource groups queues better - if I create it now, will it persist are they global? And how do we edit it? What are the parameters I can play around? How do they change with 2CN? For ingestions, and queries? @Rocky
r
Hi there! I'm Rocky, and I'd be happy to help you dive into StarRocks Resource Groups. Resource Groups are the primary way we handle resource isolation and workload management. Here is the breakdown of how they work, how they scale with your 2 CN nodes, and how you can manage them: 1. Persistence and Scope * Persistence: Yes, they are persistent. Once you create a resource group using 
CREATE RESOURCE GROUP
, the definition is stored in the FE (Frontend) metadata. It will survive restarts. * Global Scope: The definition is global (managed by the FE), but the resource limits (CPU and Memory) are enforced per-node. * If you set a 
mem_limit = 0.2
, it means each of your 2 CNs will allow that resource group to use up to 20% of its individual query memory pool. * The 
concurrency_limit
is managed by the FE across the whole cluster for that specific group. 2. How to Edit Them You can modify an existing resource group at any time using the 
ALTER RESOURCE GROUP
statement. For example:
Copy code
sql
ALTER RESOURCE GROUP my_group 
WITH (
    'cpu_weight' = '10',
    'mem_limit' = '0.5',
    'concurrency_limit' = '20'
);
Changes take effect almost immediately (usually within seconds) across the cluster. 3. Parameters You Can "Play Around" With The most important parameters are: * 
cpu_weight
(Soft Limit): Determines the relative share of CPU. If other groups are idle, this group can "borrow" more. (Replaces 
cpu_core_limit
in newer versions). * 
exclusive_cpu_cores
(Hard Limit): Reserves specific cores just for this group. Other groups cannot use these cores even if this group is idle. * `mem_limit`: The percentage (0 to 1.0) of the node's query memory pool available to this group. * `concurrency_limit`: The maximum number of concurrent queries allowed. * `big_query_cpu_second_limit`: Automatically kills a query in this group if it consumes more than X CPU seconds on a single node. * `big_query_scan_rows_limit`: Limits how many rows a single query can scan. 4. Behavior with 2 CNs (Scaling) When you have 2 CNs (or BEs): * Queries: If you have a 
concurrency_limit
of 10, the FE ensures only 10 queries for that group run simultaneously across the cluster. The CPU/Memory limits apply to the fragments of those queries running on each of the 2 CNs. * Ingestions: You can create a group specifically for 
INSERT
tasks (like Stream Load or Broker Load). * Use the 
query_type
classifier: 
TO (query_type IN ('insert'))
. * This ensures that a heavy data load doesn't starve your analytical 
SELECT
queries of CPU or memory. 5. Managing the "Queue" By default, if a resource group hits its 
concurrency_limit
, new queries will fail with an error. To make them queue instead of failing, you need to enable the query queue: 1. Set the global variable: 
SET GLOBAL enable_group_level_query_queue = true;
2. Configure queue parameters in the resource group like 
max_queued_queries
and 
queued_timeout_ms
. Pro-Tip for Rocky Users: To see if your queries are actually hitting the right group, you can check the 
information_schema.resource_group_usage
(for real-time) or look at the 
ResourceGroup
field in your Audit Log. Do you have a specific workload in mind (e.g., high-frequency small inserts vs. giant complex joins) that you're trying to tune for? I can give you some specific parameter recommendations! References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md
r
@Rocky What are the defaults for big_query* parameters? what if we don't want them to be in effect? I.e we dont want long running queries to fail and we don't want to enforce scan rows limit?
r
In StarRocks, the 
big_query*
parameters are primarily used within Resource Groups to identify and restrict resource-intensive queries, or as session variables to control logging/profiling. 1. Resource Group Parameters (Limits) These parameters are properties of a resource group and are designed to terminate queries that exceed specific resource thresholds on any single BE node. Parameter | Default Value | Description 
big_query_cpu_second_limit
 | 
0
| The maximum CPU time (in seconds) a query can use on each BE node. 
big_query_scan_rows_limit
 | 
0
| The maximum number of rows a query can scan on each BE node. 
big_query_mem_limit
 | 
0
| The maximum memory (in bytes) a query can use on each BE node. How to disable them: * The default is 
0
, which means these limits are not in effect (unlimited). * If you have a resource group where these are set to a positive value and you want to disable the enforcement, you should set them back to 
0
using the 
ALTER RESOURCE GROUP
command:
Copy code
sql
    ALTER RESOURCE GROUP your_group_name WITH (
        'big_query_cpu_second_limit' = '0',
        'big_query_scan_rows_limit' = '0',
        'big_query_mem_limit' = '0'
    );
2. System Variables (Profiling & Logging) There are also variables that use the 
big_query
prefix to control observability: * 
big_query_profile_threshold
: * Default: 
0
(or 
""
in some versions, effectively disabled). * Effect: If set (e.g., to 
10s
), StarRocks will automatically generate a query profile if the query duration exceeds this threshold, even if 
enable_profile
is set to 
false
. * Does it fail queries? No. It only triggers profile generation for troubleshooting. * 
big_query_log_modules
 (FE Config): * Default: 
{"query"}
. * Effect: Controls which modules log into the 
big_query.log
file. This does not affect query execution or success. Important: Long-Running Queries If your long-running queries are failing but your 
big_query_cpu_second_limit
is already 
0
, the failure is likely caused by the 
query_timeout
 session variable. * 
query_timeout
: * Default: 
300
seconds. * Effect: This is a wall-clock limit. If a query runs longer than this, it is killed. * To disable/extend: Set it to a very large value (e.g., 
86400
for 24 hours).
Copy code
sql
    SET GLOBAL query_timeout = 86400; -- For all new sessions
    -- OR --
    SET query_timeout = 86400; -- For the current session
Summary for your request: To ensure "big query" limits are not in effect: 1. Ensure 
big_query_cpu_second_limit
, 
big_query_scan_rows_limit
, and 
big_query_mem_limit
are set to 
0
in your resource groups (this is the default). 2. Increase your 
query_timeout
session variable, as this is the most common reason for long-running queries to fail. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • administration/management/monitoring/metrics.md
r
Do you have a specific workload in mind (e.g., high-frequency small inserts vs. giant complex joins) that you're trying to tune for? I can give you some specific parameter recommendations!
@Rocky Yes, We will have short bursts of inserts, and then in parallel, or with some delay, complex joins. We will also have intermittent queries to export data etc
r
That is a classic "Hybrid" workload! Handling short bursts of inserts while running complex joins requires a balance between write throughput (compaction) and query resource isolation. Here are the specific parameter recommendations for your StarRocks cluster, categorized by the workload components you mentioned: 1. Tuning for Short Bursts of Inserts To prevent "short bursts" from creating too many small files (which degrades join performance later), you need to tune the compaction process to keep up. * Table Type: Use Primary Key tables if your inserts involve updates or deletes. They are optimized for high-frequency writes. Use Duplicate Key tables for append-only logs. * BE Configuration (
be.conf
): * `compact_threads`: Increase this to allow more parallel compaction. A good rule of thumb is 25% of your total CPU cores (e.g., if you have 32 cores, set to 8). * `max_cumulative_compaction_num_singleton_deltas`: Set this to 100 (default is often higher). This makes each compaction task smaller and faster, reducing the "compaction score" quickly during bursts. * `write_buffer_size`: Increase to 256MB or 512MB (default 100MB) if your bursts involve large amounts of data, to reduce the frequency of flushing small files to disk. * Frontend (FE) Configuration: * If using Shared-Data (Cloud-native): Set 
lake_compaction_max_tasks
to 
-1
(adaptive) or a higher fixed number to trigger more concurrent tasks. 2. Tuning for Complex Joins Since you have parallel inserts, you must protect your join performance from being "starved" of CPU or memory. * Resource Groups (Crucial): Create separate resource groups to isolate the inserts from the joins. This prevents a heavy join from blocking an insert, and vice-versa.
Copy code
sql
    -- Example: Create a group for complex joins
    CREATE RESOURCE GROUP 'analytics_group'
    PROPERTIES (
        "cpu_weight" = "10",
        "mem_limit" = "0.6", -- Allocate 60% of BE memory
        "concurrency_limit" = "20"
    );
* Spill to Disk: For very complex joins that might hit memory limits, enable spilling: * Set 
enable_spill = true
in your session or resource group. * Pipeline Engine: Ensure 
enable_pipeline_engine = true
(default in 3.x) is on, as it handles parallel join execution much more efficiently than the old engine. 3. Tuning for Intermittent Exports Exports can be resource-intensive because they scan large amounts of data and serialize it. * Export FE Parameters: * `export_task_pool_size`: Keep this at a low value (e.g., 3-5) so that export tasks don't take over all available threads during your join windows. * `export_max_bytes_per_be_per_task`: Set to 256MB or 512MB. This controls how StarRocks splits the export job into smaller tasks. * Resource Group Classification: Assign your export user or query type to a specific resource group with a lower 
cpu_weight
so it runs in the background without impacting the complex joins.
Copy code
sql
    -- Classify inserts and exports separately
    CREATE RESOURCE GROUP 'load_group' PROPERTIES ("cpu_weight" = "2", "mem_limit" = "0.2");
    ALTER RESOURCE GROUP 'load_group' ADD CLASSIFIER (query_type = 'insert');
Summary Checklist: 1. Batch your inserts: Even with "bursts," try to avoid 
INSERT INTO ... VALUES (1,2,3)
one by one. Use 
Stream Load
or batch inserts of at least 100+ rows. 2. Monitor Compaction: Run 
SHOW PROC '/compactions';
during a burst. If the "Score" is consistently > 100, increase your 
compact_threads
. 3. Memory Limits: Monitor 
load_process_max_memory_limit_percent
(default 30%) to ensure your insert bursts don't consume all the memory needed for your complex joins. Does your "short burst" usually come from a specific source like Kafka, or are they manual SQL inserts? I can give you the exact 
Stream Load
or 
Routine Load
settings if needed! References
r
@Rocky how does 
mem_limit
work in resource group queues?
r
In StarRocks, the 
mem_limit
parameter in a resource group works as a hard limit for execution and a reference for spilling, rather than a direct "queuing" trigger like concurrency. Here is a breakdown of how it functions within the resource management and queuing system: 1. Hard Execution Limit 
mem_limit
specifies the percentage of the BE's query memory pool (by default, 90% of BE RAM) that a resource group can consume. * Per-BE Enforcement: This limit is enforced on each BE node independently. * Failure on Overrun: If the total memory used by all running queries in a resource group exceeds this limit on a specific BE, new memory allocations will fail, and the query will typically be terminated with a 
Memory limit exceeded
error. * No Native Queuing: Unlike 
concurrency_limit
, exceeding the 
mem_limit
during execution does not put the query into a queue; it results in a failure. 2. Interaction with Query Queues StarRocks has two types of queues where memory plays different roles: * Global Query Queue: You can set a memory threshold for the entire BE using 
query_queue_mem_used_pct_limit
. If the total BE memory exceeds this percentage, all new incoming queries (regardless of resource group) are queued until memory usage drops. * Resource Group Queue: Triggered when 
enable_group_level_query_queue
is set to 
true
. However, the queuing triggers for resource groups are specifically 
concurrency_limit
and 
max_cpu_cores
. The 
mem_limit
is not currently used as a pre-check trigger to "queue" a query before it starts; it is enforced once the query begins running. 3. Memory Spilling (v3.1.7+) The 
mem_limit
works with the 
spill_mem_limit_threshold
to prevent query failures: _ If a resource group's memory usage exceeds 
mem_limit _ spill_mem_limit_threshold
, StarRocks will begin spilling intermediate data to disk for queries within that group. * This allows the group to continue processing even when it is close to its 
mem_limit
, effectively "stretching" the available memory at the cost of disk I/O. 4. Shared Memory Pools (v4.0+) Using the 
mem_pool
parameter, you can group multiple resource groups together to share a single 
mem_limit
. * Example: If 
RG1
and 
RG2
are both in 
shared_pool
with a 
mem_limit
of 
50%
, they collectively can use up to 50% of the BE query memory. If 
RG1
uses 40%, 
RG2
only has 10% remaining before reaching the limit. Summary Table Feature | Triggered By | Result Queuing (Global) | 
query_queue_mem_used_pct_limit
| Queries wait in a global queue. Queuing (Group) | 
concurrency_limit
, 
max_cpu_cores
| Queries wait in a group-specific queue. Spilling | 
mem_limit * spill_mem_limit_threshold
| Data spills to disk to save memory. Execution Limit | 
mem_limit
| Query fails if exceeded during runtime. References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md • administration/management/resource_management/query_queues.md
j
@Rocky what dimensions can er segment resource groups on? Can we add tags at query time or does it need to be user based?
r
In StarRocks, resource groups can be segmented using Classifiers. These classifiers allow you to automatically route queries to specific resource groups based on various dimensions. Dimensions for Segmenting (Classifiers) You can define one or more conditions in a classifier to match query properties: * 
user
: The name of the user who initiated the query. * 
role
: The role of the user. * 
query_type
: The type of query (supported values are 
SELECT
and 
INSERT
). * 
source_ip
: The CIDR block (IP range) from which the query is initiated. * 
db
: The database being accessed by the query. * 
plan_cpu_cost_range
: The system-estimated CPU cost of the query (available from v3.1.4 onwards). * 
plan_mem_cost_range
: The system-estimated memory cost of the query (available from v3.1.4 onwards). ────────── Applying Resource Groups at Query Time While the classifiers mentioned above are the primary way to automate routing (based on user, DB, etc.), you can also specify or override the resource group at query time or session time. 1. Session/Query Level Override: Instead of relying purely on user-based classifiers, you can manually set the resource group for your current session. This is effectively "at query time" because it overrides the default classifier logic:
Copy code
sql
    -- Directs all subsequent queries in this session to 'my_resource_group'
    SET resource_group = 'my_resource_group';
2. Does it need to be user-based? No, it does not have to be strictly user-based. You can route based on the Source IP, the Database being queried, or even the estimated cost of the query. 3. Query Tags: StarRocks does not currently support a "tag" as a standard dimension within the classifier definition (i.e., you can't create a classifier like 
TO (tag='analytics')
). However, using the 
SET resource_group
command before a query is the standard way to achieve dynamic, per-query routing. Example of creating a group with multiple dimensions:
Copy code
sql
CREATE RESOURCE GROUP analytic_group
TO 
    (user='alice', role='analyst'), -- User/Role based
    (db='sales_reports'),           -- Database based
    (source_ip='192.168.1.0/24')    -- Network based
WITH (
    "cpu_weight" = "10",
    "mem_limit" = "20%"
);
References • administration/management/resource_management/resource_group.md
n
@Rocky I have a requirements: A starrocks clusters build for products, there has two products,How to config the resource group?
r
To configure resource groups for a StarRocks cluster supporting two different products, you should follow these steps to ensure workload isolation and resource management. 1. Conceptual Design You need to decide how to distinguish queries from the two products. The most common ways are by Database or by User. * Product A: Uses 
db_a
or logs in with user 
user_a
. * Product B: Uses 
db_b
or logs in with user 
user_b
. 2. Enable Resource Isolation Resource groups require the Pipeline Engine (enabled by default in StarRocks 3.1.0+). To be safe, ensure it is enabled globally:
Copy code
sql
SET GLOBAL enable_pipeline_engine = true;
3. Create Resource Groups You can choose between two types of CPU limits: * 
cpu_weight
(Shared): Flexible. If one product is idle, the other can use more CPU. If both are busy, they get CPU based on the weight ratio. * 
exclusive_cpu_cores
(Exclusive): Hard limit. The cores are reserved solely for that group. Example: Isolation based on Database If you want Product A to have 60% and Product B to have 40% of resources:
Copy code
sql
-- Resource Group for Product A
CREATE RESOURCE GROUP product_a_group
TO (db='db_product_a')
WITH (
    "cpu_weight" = "60",
    "mem_limit" = "50%",
    "concurrency_limit" = "100"
);

-- Resource Group for Product B
CREATE RESOURCE GROUP product_b_group
TO (db='db_product_b')
WITH (
    "cpu_weight" = "40",
    "mem_limit" = "30%",
    "concurrency_limit" = "50"
);
Example: Isolation based on User If Product A and B use specific database users:
Copy code
sql
CREATE RESOURCE GROUP product_a_group
TO (user='prod_a_user')
WITH (
    "cpu_weight" = "10",
    "mem_limit" = "40%"
);
4. Key Parameters Explained * 
TO
(Classifiers): This is how StarRocks knows which query belongs to which group. You can use 
user
, 
role
, 
db
, or 
source_ip
. * `cpu_weight`: A relative value compared to other groups. * `mem_limit`: The maximum percentage of memory on each BE node this group can use. * `concurrency_limit`: Limits the number of concurrent queries to prevent a single product from overwhelming the cluster. * `big_query_mem_limit`: (Optional) You can add this to kill "rogue" queries that exceed a certain memory threshold. 5. Managing Resource Groups * View all groups: 
SHOW RESOURCE GROUPS ALL;
* Update a group:
Copy code
sql
    ALTER RESOURCE GROUP product_a_group WITH ("cpu_weight" = "80");
* Delete a group: 
DROP RESOURCE GROUP product_a_group;
Best Practices for Production 1. Monitor Usage: Use the 
Information_schema.resource_group_usage
(if available in your version) or audit logs to see if your limits are too restrictive. 2. Default Group: Queries that don't match any classifier will fall into the 
default_wg
. Ensure your important products have specific classifiers so they don't compete with "unknown" traffic. 3. Small Queries vs Big Queries: If one product has both dashboarding (fast) and ETL (slow) queries, you can create two groups for the same product using the 
query_type
classifier (e.g., 
query_type in ('select')
vs 
query_type in ('insert')
). References • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md • administration/management/resource_management/resource_group.md
n
@Rocky IF add another product, how to config the resource?
r
Adding a third product to your StarRocks cluster follows the same pattern as the first two, but you might need to rebalance the weights of the existing products to ensure fair resource distribution. Here is the 3-step process to add and configure resources for a new product: 1. Identify the New Product's Workload Determine how to identify Product C's queries (e.g., a new database 
db_product_c
or a new user 
user_c
) and what its priority is relative to the others. 2. Create the Resource Group for Product C If you want Product C to have a specific share of resources, create its group. Example: Shared Resource Mode (Using 
cpu_weight
) In this mode, if the cluster is busy, CPU is distributed based on the ratio of weights.
Copy code
sql
CREATE RESOURCE GROUP product_c_group
TO (db='db_product_c') -- or (user='user_c')
WITH (
    "cpu_weight" = "30",
    "mem_limit" = "20%",
    "concurrency_limit" = "50"
);
3. Rebalance Existing Groups (Important) When you add a third product, the "pie" gets split into more pieces. You should use 
ALTER RESOURCE GROUP
to adjust the original products if their current weights/limits are now too high. Scenario: You have 100% capacity to split. * Old Setup: Product A (60), Product B (40). Total = 100. * New Setup: You want a 40/30/30 split.
Copy code
sql
-- Adjust Product A down from 60 to 40
ALTER RESOURCE GROUP product_a_group WITH ("cpu_weight" = "40", "mem_limit" = "40%");

-- Adjust Product B down from 40 to 30
ALTER RESOURCE GROUP product_b_group WITH ("cpu_weight" = "30", "mem_limit" = "30%");

-- Product C (created in step 2) takes the remaining 30
Key Considerations for Scaling to Multiple Products: * Weight is Relative: 
cpu_weight
is not a percentage. If Group A has weight 10 and Group B has weight 10, they split resources 50/50. If you add Group C with weight 10, they all automatically split resources 33/33/33. * Memory Sums: The 
mem_limit
is a hard limit for that specific group. Ensure the sum of 
mem_limit
across all groups (including the 
default_wg
) is managed carefully. While StarRocks allows the sum to exceed 100%, it's best practice to keep the total around 80-90% to leave room for system tasks. * Exclusive Cores: If Product C is a "Critical Dashboard" that must never be slow, consider using 
exclusive_cpu_cores
instead of 
cpu_weight
. This reserves physical CPU cores that other products cannot touch. _ _Note: Only use this for highly critical, small-scale workloads.* How to Check the New Balance After adding the third product, run this to see the current distribution of all products:
Copy code
sql
SHOW RESOURCE GROUPS ALL;
This will show you the weights, memory limits, and the assigned classifiers (databases/users) for all three products side-by-side. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md
n
@Rocky The total cpu_weight and mem_limit of products are 100%, If I set the default_wg, how to config the resource ?
r
When you have multiple products whose resource requirements total 100%, you must account for the 
default_wg
. This group is a "catch-all" for any query that doesn't match your product-specific classifiers (such as queries from the 
admin
user, internal system tasks, or accidental queries against other databases). By default, 
default_wg
has a very high priority (
cpu_weight
equals the number of CPU cores and 
mem_limit
is 100%). If you don't configure it, it could potentially starve your products of resources. Here is how to configure it correctly: 1. The "Safety Buffer" Strategy (Recommended) In this approach, you allocate a small "slice" of the cluster to the 
default_wg
to ensure the system remains stable and manageable, while giving the bulk of resources to your products. Example Allocation: * Product A: 40% CPU / 40% Mem * Product B: 30% CPU / 25% Mem * Product C: 25% CPU / 25% Mem * default_wg: 5% CPU / 10% Mem (Safety buffer) 2. Configuration SQL You can modify the 
default_wg
just like any other group using 
ALTER RESOURCE GROUP
. Note that you cannot add classifiers to it; it always catches everything else.
Copy code
sql
-- 1. Adjust your existing products to make room (Total 95%)
ALTER RESOURCE GROUP product_a_group WITH ("cpu_weight" = "40", "mem_limit" = "40%");
ALTER RESOURCE GROUP product_b_group WITH ("cpu_weight" = "30", "mem_limit" = "25%");
ALTER RESOURCE GROUP product_c_group WITH ("cpu_weight" = "25", "mem_limit" = "25%");

-- 2. "Shrink" the default group so it doesn't take over
-- Set weight to a low number (e.g., 5) and limit its memory
ALTER RESOURCE GROUP default_wg WITH (
    "cpu_weight" = "5", 
    "mem_limit" = "10%"
);
3. Understanding the Mechanics * CPU Weight is Relative: The 
cpu_weight
is a ratio. If the total of all weights is 100 (40+30+25+5), then a weight of 
5
for 
default_wg
means it gets exactly 5% of CPU cycles when the cluster is under full load. If the products are idle, the 
default_wg
can still use more. * Memory Limit is a Hard Cap: 
mem_limit
is a percentage of the total memory on each BE. If the sum of all 
mem_limit
exceeds 100% (over-provisioning), StarRocks allows it, but you risk the Backend (BE) process hitting its global limit and killing queries if everyone tries to use their full limit at once. It is best to keep the sum around 90-100%. * Don't set 
mem_limit
to 0% for `default_wg`: You still need to run basic metadata queries or maintenance tasks. Setting it too low (e.g., <5%) might cause system queries to fail with "Memory limit exceeded." 4. How to Verify After applying the changes, run this command to see the new "pie chart" of your cluster:
Copy code
sql
SHOW RESOURCE GROUPS ALL;
Check that the 
cpu_weight
and 
mem_limit
columns align with your plan across your products and the 
default_wg
. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md
n
@Rocky Do you think set the big-query limit resource for each product is good ?
r
Yes, setting big-query limits for each product is highly recommended and is considered a best practice for production environments. It acts as a "safety fuse" to prevent a single poorly written query from crashing a node or starving other products of resources. Here is why it is good and how to implement it effectively: 1. Why it is a Good Practice * Blast Radius Control: If Product A accidentally runs a massive 
CROSS JOIN
without a filter, the big-query limit will kill that specific query once it hits the threshold, rather than letting it consume 100% of the Backend (BE) memory and causing an Out-of-Memory (OOM) crash for the whole node. * Different SLAs: You can set different "patience" levels for different products. * Dashboard Product: Might have a strict 20s CPU limit because users expect fast results. * Data Science Product: Might have a 300s CPU limit because they run complex analytical jobs. * Predictable Performance: It ensures that one "heavy" product cannot hog the shared resources (CPU/Scan rows) indefinitely. 2. The 3 Key "Big Query" Parameters You can add these to your 
CREATE
or 
ALTER RESOURCE GROUP
statements: Parameter | Description 
big_query_cpu_second_limit
 | Max CPU time (in seconds) a query can use on a single BE. 
big_query_scan_rows_limit
 | Max number of rows a query can scan on a single BE. 
big_query_mem_limit
 | Max memory (in bytes) a query can use on a single BE. 3. Practical Example Let's say Product A is a user-facing app (needs protection from slow queries) and Product B is an internal reporting tool (allows heavier queries).
Copy code
sql
-- Product A: Strict limits (Kill queries > 30s CPU or > 5GB RAM per BE)
ALTER RESOURCE GROUP product_a_group WITH (
    "big_query_cpu_second_limit" = "30",
    "big_query_mem_limit" = "5368709120", -- 5GB
    "big_query_scan_rows_limit" = "100000000" -- 100M rows
);

-- Product B: Relaxed limits (Kill queries > 5 min CPU or > 20GB RAM per BE)
ALTER RESOURCE GROUP product_b_group WITH (
    "big_query_cpu_second_limit" = "300",
    "big_query_mem_limit" = "21474836480" -- 20GB
);
4. Important Implementation Tips 1. Limits are Per-Node: Remember that 
big_query_mem_limit
= 5GB means 5GB on each BE. If you have 3 BEs, the query could technically use up to 15GB total across the cluster. 2. Start High, then Tighten: If you aren't sure what the limits should be, start with very high values (or 0 to disable). Monitor your 
fe.audit.log
or 
information_schema
to see typical query usage, then set the limit slightly above the 99th percentile of your "normal" queries. 3. Default Group Protection: Always set big-query limits on your 
default_wg
. This prevents unauthorized users or accidental "un-classified" queries from impacting your production products. Summary Setting these limits is the difference between "one user ran a bad query and the whole cluster is slow" and "one user ran a bad query and only they got an error message." For 'Exceeding Big Query Limit'."* It'."* It'."* It'."* It'."* It'."* It'."* Iimit'."* It'."* It'."* It'."** References • administration/management/monitor_manage_big_queries.md • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md • administration/management/monitor_manage_big_queries.md • administration/management/resource_management/resource_group.md
n
@Rocky For product A, Should i set bigquery limit and shour query limit, so Each product has two resource groups ?
r
In StarRocks, whether you should create one or two resource groups for a product depends on your goal: stopping bad queries versus protecting fast queries. Here is the breakdown of the two strategies: Option 1: One Resource Group (Simpler) Goal: Prevent "runaway" queries from crashing the cluster. In this setup, you have one group for Product A and you set the 
big_query_..._limit
parameters. If a query exceeds these, it is killed. * Best for: General protection where you don't mind if a medium-sized query uses the same resources as a small query. * Configuration:
Copy code
sql
    ALTER RESOURCE GROUP product_a_group WITH (
        "big_query_cpu_second_limit" = "100", -- Kills query if > 100s CPU
        "big_query_mem_limit" = "1073741824"  -- Kills query if > 1GB RAM
    );
Option 2: Two Resource Groups (Advanced Isolation) Goal: Ensure that a long-running analytical query doesn't make a 100ms dashboard query take 5 seconds. You can use Classifiers based on the estimated cost of a query (
plan_cpu_cost_range
) to automatically route queries to a "Fast Lane" or a "Batch Lane." * Best for: Products with mixed workloads (e.g., a dashboard and a data export tool using the same user/DB). * How it works: 1. Product A - Short Group: High 
cpu_weight
, low 
concurrency_limit
, and strict 
big_query
limits. 2. Product A - Long Group: Lower 
cpu_weight
(so it doesn't starve the short queries) and relaxed 
big_query
limits. Example Configuration:
Copy code
sql
-- 1. Create the "Fast Lane" for Product A
CREATE RESOURCE GROUP product_a_short
TO (db='db_a', plan_cpu_cost_range='(0, 1000000]') -- Routes small queries here
WITH (
    "cpu_weight" = "80", 
    "big_query_cpu_second_limit" = "10" -- Kill if it accidentally runs long
);

-- 2. Create the "Batch Lane" for Product A
CREATE RESOURCE GROUP product_a_long
TO (db='db_a', plan_cpu_cost_range='(1000000, -1]') -- Routes heavy queries here
WITH (
    "cpu_weight" = "20", 
    "big_query_cpu_second_limit" = "3600" -- Allow up to 1 hour
);
Which should you choose? * Start with Option 1 (One Group): It is much easier to manage. Most users find that setting the 
big_query_..._limit
on a single group is enough to keep the cluster stable. * Move to Option 2 (Two Groups) ONLY IF: You notice that when Product A runs a "big" query, the "small" queries for the same product become noticeably slower. Key Performance Note: From v3.3.5 onwards, if you have a "Short Query" group that must be fast, you should use 
exclusive_cpu_cores
instead of 
cpu_weight
. This reserves physical CPU cores for that group so they are never "borrowed" by other products. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • administration/management/monitor_manage_big_queries.md References • administration/management/resource_management/resource_group.md
n
@Rocky What's your recommend?
r
My recommendation for managing your products in StarRocks depends on the nature of the workload within each product. For most users, I recommend starting with Strategy 1 and only moving to Strategy 2 if you have specific performance issues. ────────── Strategy 1: One Resource Group per Product (Recommended for 90% of cases) Instead of splitting every product into "Short" and "Long" groups, create one robust group per product. This keeps your configuration clean and easy to monitor. * Why: It is simpler to maintain. You can still protect the cluster by setting 
big_query
limits within that single group. If a query is truly "bad," it gets killed. If it’s just "heavy," it runs but is throttled by the product's overall 
cpu_weight
. * Key Settings: * 
cpu_weight
: Set this to represent the product's priority relative to others. * 
big_query_cpu_second_limit
: Set this to a "sanity check" value (e.g., 300s). This prevents a single product from spinning the CPU for an hour on a bad query. ────────── Strategy 2: Two Resource Groups per Product (Recommended for High SLA / Mixed Workloads) Use this only for products that have both high-concurrency/fast dashboard queries and heavy data-science/reporting queries coming from the same user or database. * Why: To prevent a "Line-of-Blockage." In a single group, a massive query might not kill the cluster, but it could make your 100ms dashboard query take 2 seconds because they are competing for the same slice of the product's CPU weight. * How to implement: Use the 
plan_cpu_cost_range
classifier to automatically route queries. * Group A (Fast): 
plan_cpu_cost_range = (0, 1000000]
. Give it a high 
cpu_weight
or even 
exclusive_cpu_cores
(if on v3.3.5+). * Group B (Batch): 
plan_cpu_cost_range = (1000000, -1]
. Give it a lower 
cpu_weight
. ────────── The "Rocky" Recommended Blueprint If I were setting up your cluster today, here is the specific configuration I would use for Product A: Goal | Parameter | Value | Reason Stability | 
big_query_mem_limit
| 20-30% of BE RAM | Prevents "Out of Memory" crashes. Stability | 
big_query_cpu_second_limit
| 300 | Prevents "Infinite Loop" or poorly written queries. SLA | 
exclusive_cpu_cores
| 2 - 4 | (v3.3.5+) Guarantees a "fast lane" that other products cannot touch. Concurrency | 
concurrency_limit
| 100 | Prevents "Query Flooding" from a single product. Summary Checklist: 1. Don't over-complicate: Start with one group per product. 2. Always set limits: Put 
big_query_mem_limit
and 
big_query_cpu_second_limit
on every group, including 
default_wg
. 3. Monitor first: Use the 
fe.audit.log
or the 
information_schema.be_audit_log
to see the actual 
cpuCostNs
and 
memUsageBytes
of your products before tightening the limits. Pro-tip: If you use StarRocks v3.3.5 or later, use 
exclusive_cpu_cores
for your most important product's "short query" group. This provides "hard isolation," ensuring those queries are never delayed by other workloads. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • best_practices/audit_log_resource_group.md References • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md • administration/management/resource_management/resource_group.md • best_practices/audit_log_resource_group.md
n
@Rocky Apply * Product A: 40% CPU / 40% Mem * Product B: 30% CPU / 25% Mem * Product C: 25% CPU / 25% Mem * default_wg: 5% CPU / 10% Mem (Safety buffer) ; And Set big_query_mem_limit
r
To apply this configuration, you will use the 
CREATE RESOURCE GROUP
(for new products) and 
ALTER RESOURCE GROUP
(for the existing 
default_wg
) commands. 1. Apply Configuration for Products A, B, and C For CPU percentages, we set the 
cpu_weight
proportionally (40, 30, 25, 5). For Memory, we use the 
mem_limit
parameter with a percentage string.
Copy code
sql
-- Product A: 40% CPU, 40% Memory
CREATE RESOURCE GROUP product_a_group
TO (db='db_a') -- Replace with your actual classifier (user, role, or db)
WITH (
    "cpu_weight" = "40",
    "mem_limit" = "40%",
    "big_query_mem_limit" = "10737418240" -- Example: 10GB limit per query (in bytes)
);

-- Product B: 30% CPU, 25% Memory
CREATE RESOURCE GROUP product_b_group
TO (db='db_b')
WITH (
    "cpu_weight" = "30",
    "mem_limit" = "25%",
    "big_query_mem_limit" = "5368709120" -- Example: 5GB limit per query
);

-- Product C: 25% CPU, 25% Memory
CREATE RESOURCE GROUP product_c_group
TO (db='db_c')
WITH (
    "cpu_weight" = "25",
    "mem_limit" = "25%",
    "big_query_mem_limit" = "5368709120"
);
2. Configure 
default_wg
(Safety Buffer) Since 
default_wg
is a system-defined group, you must use 
ALTER
. This group captures any queries that don't match the classifiers of Products A, B, or C.
Copy code
sql
ALTER RESOURCE GROUP default_wg WITH (
    "cpu_weight" = "5",
    "mem_limit" = "10%",
    "big_query_mem_limit" = "2147483648" -- Example: 2GB limit for "unknown" queries
);
Important Technical Details: 1. 
cpu_weight
vs Percentage: StarRocks uses a weight system. Since your requested values (40+30+25+5) add up to exactly 100, the weight effectively acts as a percentage of the total CPU capacity when all groups are active. 2. 
big_query_mem_limit
is in Bytes: Unlike 
mem_limit
, this parameter requires a numeric value in bytes. * 
1073741824
= 1 GB * 
5368709120
= 5 GB * 
10737418240
= 10 GB 3. Per-Node Limits: These limits are enforced on each Backend (BE) node. If you set a 
big_query_mem_limit
of 10GB and you have 3 BEs, a distributed query could technically consume up to 30GB of memory across the cluster before being killed. 4. Memory Sum: StarRocks allows the sum of 
mem_limit
across all groups to exceed 100%. However, in your case, it equals 100%, which is a safe and "hard" allocation strategy. Recommendation for 
big_query_mem_limit
value: You should set 
big_query_mem_limit
to a value slightly lower than the total memory available to that resource group in bytes. For example, if your BE has 100GB of RAM and Product A has a 40% 
mem_limit
(40GB), setting 
big_query_mem_limit
to 20GB or 30GB ensures one query doesn't starve other queries within the same product. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md • faq/resource_isolation_faq.md References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md • faq/resource_isolation_faq.md • administration/management/monitor_manage_big_queries.md • administration/management/monitor_manage_big_queries.md
n
@Rocky So, If there have not query match default_wg, there still have 10% mem will be waste
r
Actually, no memory will be wasted, even if the 
default_wg
is idle. In StarRocks, the 
mem_limit
for a resource group acts as a hard ceiling (quota), but it is not a physical reservation. Here is how it works: 1. Limits are Quotas, not Reservations Setting a 10% memory limit for 
default_wg
does not physically carve out 10% of your RAM and hide it from other groups. It simply means that if a query lands in the default group, it cannot use more than 10% of the BE's query memory. If no queries are running in 
default_wg
, that physical RAM remains available for the system and other resource groups to use (up to their own respective limits). 2. You Can Over-Subscribe (Total > 100%) StarRocks allows the sum of all 
mem_limit
values to exceed 100%. This is the best way to prevent "waste." If you want your products to be able to use extra memory when the cluster is quiet, you should set their limits slightly higher. For example: * Product A: 60% (instead of 40%) * Product B: 50% (instead of 30%) * Product C: 40% (instead of 25%) * default_wg: 20% (instead of 5%) Why do this? * If only Product A is running, it can now use up to 60% of the RAM, which is better for performance. * If all products start running at the same time and hit their limits, StarRocks' internal memory tracker will manage the pressure. However, queries will only fail if they exceed their own group limit or if the total BE memory usage hits the process limit (usually 90% of RAM). 3. Recommendation for your setup If you want to stick to your original plan but ensure no capacity is "locked away," you can simply set the 
default_wg
higher (e.g., 50% or even 100%) and leave your product groups at your desired percentages. Revised Recommendation: To keep things safe but flexible, I suggest: 1. Set Product limits to what you want them to "guarantee" themselves. 2. Over-subscribe the total. If Product A is your most important product, give it a 
mem_limit
of 
80%
. This way, if Products B and C are idle, Product A can use almost the whole machine. It will only be restricted to its "fair share" if other groups are also active and competing for RAM. Key Difference: CPU vs. Memory * CPU (
cpu_weight
): This is "soft." If Product A is idle, its 40% CPU weight is automatically and instantly shared among B and C. No configuration is needed to avoid "waste." * Memory (
mem_limit
): This is "hard." A group cannot exceed its limit even if the rest of the machine is empty. To avoid waste here, you must over-subscribe the percentages. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md • best_practices/audit_log_resource_group.md
n
@Rocky 可以设置
Copy code
query_queue_options
吗
r
在 StarRocks 中，*并没有直接名为 
query_queue_options
的资源组属性（Property）*。 你可能是在查阅较新版本（如 v3.3+）的内部配置或 FE 配置项时看到了相关的术语。在 StarRocks 的标准 SQL 语法中，针对资源组的查询队列配置，通常是通过以下*参数*和*系统变量*来完成的： 1. 资源组内的队列相关参数 在 
CREATE RESOURCE GROUP
或 
ALTER RESOURCE GROUP
的 
WITH
子句中，你可以直接设置触发排队的阈值： * 
concurrency_limit
: 限制该资源组在单个 BE 上的最大并发查询数。超过此限制的查询会进入队列（如果启用了组级队列）。 * 
max_cpu_cores
 (v3.1.4+): 限制该资源组在单个 BE 上可使用的 CPU 核心数。 示例用法：
Copy code
sql
ALTER RESOURCE GROUP product_a_group WITH (
    "concurrency_limit" = "20",  -- 超过 20 个并发即排队
    "max_cpu_cores" = "10"       -- 超过 10 核占用即排队
);
2. 必须开启的开关 要让资源组级别的排队生效，必须设置以下全局变量：
Copy code
sql
-- 1. 开启组级别的查询队列（核心开关）
SET GLOBAL enable_group_level_query_queue = true;

-- 2. 开启 SELECT 查询排队
SET GLOBAL enable_query_queue_select = true;
3. 全局队列控制变量 (类似 Options) 如果你指的 
options
是控制排队行为（如等待多久、队列多长），这些通常是通过*系统变量*全局配置的，目前不支持在单个资源组内差异化设置： 变量名 | 默认值 | 描述 
query_queue_max_queued_queries
| 1024 | 队列最大容纳查询数，超过后新查询直接报错拒绝。 
query_queue_pending_timeout_second
| 300 | 查询在队列中等待的最长时间（秒），超时则报错。 4. 关于 "Query Queue V2" (v3.3.4+ 引入) 在 StarRocks v3.3.4 及以后的 FE 配置中，确实出现了 `enable_query_queue_v2`。该版本引入了更精细的调度策略（如 `SlotSelectionStrategyV2`）。 * 如果你是在 FE 配置文件 
fe.conf
中看到了类似 
query_queue_v2_xxx
的参数，这些是 *FE 配置项*，用于调整全局调度器的行为，而不是通过 SQL 在资源组上设置的。 总结建议 如果你是想配置资源组的排队逻辑： 1. *设置阈值*：使用资源组的 `concurrency_limit`。 2. *激活功能*：确保 
enable_group_level_query_queue
已开启。 3. *调整超时/长度*：通过 
SET GLOBAL query_queue_pending_timeout_second = ...
来调整。 如果你在特定文档或代码中看到了 
query_queue_options
这个具体的 Key，请提供上下文，我可以为你确认它是否属于某个特定的 Enterprise 特性或未公开的实验性配置。 References • administration/management/resource_management/query_queues.md
n
@Rocky when resource group A and resource group B need to scramble the resource, how to control the priority?
r
In StarRocks, when multiple resource groups are "scrambling" (contending) for resources, you control their priority and resource allocation using Relative Weights (Soft Priority) or Resource Reservation (Hard Priority). There is no single "priority=1" flag; instead, priority is expressed through the following mechanisms: 1. Relative Priority (CPU Weight) The most common way to handle contention is using 
cpu_weight
. When the system is fully loaded (all groups want CPU), StarRocks distributes CPU time based on the ratio of their weights. * Group A (
cpu_weight
=80) * Group B (
cpu_weight
=20) * Contention Result: Group A will get 80% of the CPU cycles, and Group B will get 20%. * Idle Behavior: If Group A is idle, Group B can use 100% of the CPU. This ensures no "waste."
Copy code
sql
ALTER RESOURCE GROUP group_a WITH ("cpu_weight" = "80");
ALTER RESOURCE GROUP group_b WITH ("cpu_weight" = "20");
2. Hard Priority (Exclusive CPU Cores) If you want to ensure Group A always has priority regardless of how busy Group B is, you can use 
exclusive_cpu_cores
(v3.1.4+). This physically reserves a specific number of cores for that group. * Shared Groups: Run on "Shared Cores" and compete via weights. * Exclusive Groups: Have their own dedicated "Exclusive Cores." They don't have to scramble with others.
Copy code
sql
-- Reserve 8 cores exclusively for Group A
ALTER RESOURCE GROUP group_a WITH ("exclusive_cpu_cores" = "8");
_Note: Only one of 
cpu_weight
or 
exclusive_cpu_cores
can be set._ 3. Concurrency Priority (Queueing) To prevent one group from "scrambling" by launching thousands of small queries that swamp the scheduler, use 
concurrency_limit
. This forces the "lesser" group to wait in a queue before they can even begin to compete for CPU.
Copy code
sql
-- Limit Group B to 5 concurrent queries to give Group A "room"
ALTER RESOURCE GROUP group_b WITH ("concurrency_limit" = "5");
4. Memory Pressure (No "Priority") Memory is handled differently. It is a Hard Limit. There is no "priority" for memory—if Group B hits its 
mem_limit
, its queries will fail or spill to disk, regardless of whether Group A is using its memory or not. To give Group A more "priority" for memory, simply give it a much higher 
mem_limit
percentage than Group B. Summary Comparison Goal | Mechanism | Behavior Relative Importance | 
cpu_weight
| Proportional sharing under load; full use when idle. Strict Guarantee | 
exclusive_cpu_cores
| Fixed core count; zero interference from other groups. Throughput Control | 
concurrency_limit
| Limits the number of active queries to prevent cluster flooding. Safety Buffer | 
max_cpu_cores
| Limits a group to a max core count even if the machine is idle. Rocky's Tip: For a production "Product A vs Product B" scenario, I recommend setting a high 
cpu_weight
for your primary product and a lower one for the background product. This keeps the cluster efficient while protecting your VIP users during spikes. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • administration/management/resource_management/query_queues.md References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md • administration/management/resource_management/query_queues.md
n
@Rocky user 是否可以设置为多个？
r
是的，*一个资源组（Resource Group）可以设置多个 user*。 在 StarRocks 中，资源组通过 分类器（Classifier） 来识别查询并进行归类。你可以通过以下两种方式让多个用户共用同一个资源组： 1. 为资源组添加多个分类器 (推荐) 你可以在创建或修改资源组时，定义多个分类器，每个分类器指定不同的 `user`。 创建时设置：
Copy code
sql
CREATE RESOURCE GROUP my_group
TO 
    (user='alice'),
    (user='bob'),
    (user='charlie')
WITH (
    "cpu_weight" = "10",
    "mem_limit" = "20%"
);
对现有资源组添加用户：
Copy code
sql
ALTER RESOURCE GROUP my_group ADD (user='david');
────────── 2. 通过角色（Role）设置 如果你有一组用户，建议将他们归纳到一个 Role 中，然后在资源组分类器中指定该 `role`。这样，所有属于该角色的用户都会自动匹配到该资源组。
Copy code
sql
-- 将分类器设置为匹配角色
CREATE RESOURCE GROUP role_group
TO 
    (role='analyst_role')
WITH (
    "cpu_weight" = "50",
    "mem_limit" = "30%"
);
────────── 3. 注意事项 * *匹配优先级*：如果一个用户既属于 
user='alice'
的分类器，又属于 
role='admin'
的分类器，StarRocks 会计算*匹配度（Degree of Matching）*。通常 
user
的匹配权重比 
role
更具体，匹配度更高。 * *查看匹配情况*：你可以通过 
SHOW RESOURCE GROUPS ALL;
查看每个资源组下面挂载的所有分类器详情（包括 `user`、`role`、`ip` 等）。 * *DB 级设置*：你也可以通过 
db
属性让多个用户在访问特定数据库时进入同一个资源组：`(db='sales_db')`。 总结： 虽然在一个括号（分类器）里 
user
只能写一个字符串，但你可以写*无限个括号*，或者使用 Role 来批量管理用户。 参考文档 • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md References • administration/user_privs/authorization/privilege_item.md • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md • faq/resource_isolation_faq.md
Open in Slack
PreviousNext
Powered by