http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> Are there quota settings I adjust o...
# questions-and-troubleshooting
m
@Rocky Are there quota settings I adjust or implement to limit the impact of user queries on cluster health? Like a rogue user sending a massive query will not impact ingestion health.
r
Yes, StarRocks provides several mechanisms to implement quotas and limits to protect cluster health. To ensure a "rogue user" or a massive query does not impact ingestion health, you should use Resource Groups (Workload Management) and Query Queues. Here is how you can adjust these settings to isolate workloads: 1. Resource Groups (Workload Isolation) Resource groups allow you to divide cluster resources (CPU, memory, and concurrency) among different workloads. You can create a group for "Ingestion" and another for "Analysts." * Isolate by Query Type: You can create a resource group specifically for ingestion (
INSERT
tasks) and another for analytical queries (
SELECT
). * Isolate by User: Map specific users or roles to resource groups using Classifiers. Example: Creating a group for analysts with strict limits:
Copy code
sql
CREATE RESOURCE GROUP analyst_group
TO (role='analyst_role')
WITH (
    "cpu_core_limit" = "10",       -- Max CPU cores this group can use per BE
    "mem_limit" = "30%",           -- Max % of memory per BE
    "concurrency_limit" = "10",    -- Max concurrent queries for this group
    "big_query_cpu_second_limit" = "100", -- Kill query if it uses >100s of CPU time
    "big_query_scan_rows_limit" = "1000000", -- Kill query if it scans >1M rows
    "big_query_mem_limit" = "2147483648" -- Kill query if it uses >2GB RAM
);
2. Guarding Against "Rogue" Queries To prevent a single massive query from taking down a node, use the 
big_query_*
parameters within a resource group. These act as a "circuit breaker": * `big_query_cpu_second_limit`: Automatically cancels a query if its cumulative CPU time exceeds this limit. * `big_query_mem_limit`: Cancels a query if its memory consumption on a single BE exceeds this value. * `big_query_scan_rows_limit`: Limits the number of rows a query can scan. 3. Query Queuing When the cluster is under high load (high CPU or memory usage), you can enable Query Queues to buffer incoming requests instead of letting them overwhelm the system. * Global Queue: 
SET GLOBAL enable_query_queue_select = true;
* Thresholds: You can set 
query_queue_concurrency_limit
, 
query_queue_mem_used_pct_limit
, and 
query_queue_cpu_used_permille_limit
. 4. Protecting Ingestion Specifically To ensure ingestion health is prioritized over rogue user queries: 1. Define an Ingestion Resource Group: Use 
query_type in ('insert')
in the classifier. 2. Use `exclusive_cpu_cores`: If you want to guarantee resources for ingestion, you can use 
exclusive_cpu_cores
for the ingestion group. This ensures those cores are reserved and cannot be "stolen" by analytical queries. 3. Adjust `load_mem_limit`: You can set a session or global limit for memory used by loading tasks specifically via 
SET GLOBAL load_mem_limit = <bytes>;
. 5. Other Quotas * Connection Quotas: Limit the number of connections a single user can open: 
ALTER USER 'rogue_user' SET PROPERTIES ("max_user_connections" = "5");
* Storage Quotas: Limit the data volume for a database: 
ALTER DATABASE example_db SET DATA QUOTA 10T;
By combining Resource Groups (to cap the "rogue" user's impact) and Big Query Limits (to kill runaway queries), you can effectively shield your ingestion pipeline and overall cluster health. References • administration/management/monitor_manage_big_queries.md • administration/management/resource_management/Query_management.md • administration/management/resource_management/query_queues.md
m
@Rocky Should I set resource groups per BE node or is it better to provision like a single BE node just for querying? Not possible for shared nothing right? So for my 
r6in.4xlarge
instances, how much resources should I allocate users but still not impact query performance?
r
In a StarRocks Shared-Nothing architecture (the default on-prem/EC2 setup), you should use Resource Groups rather than dedicated nodes. In Shared-Nothing, data is distributed across all BE nodes. If you provisioned a "query-only" node, it would have to pull all data over the network from the data-bearing nodes for every scan, which would significantly degrade performance. Resource groups allow you to perform "Local Scans" while still enforcing strict limits. Here is how to configure your 
r6in.4xlarge
instances (16 vCPUs, 128 GiB RAM) to protect ingestion. 1. The Strategy: "Reserved Capacity" for Ingestion On a 16 vCPU machine, you want to ensure that even if a user sends a massive query, the background loading and system tasks always have breathing room. Recommended Allocation for `r6in.4xlarge`: * System/Ingestion (Reserved): 4-6 vCPUs * User Queries (Resource Group): 10-12 vCPUs 2. Implementation Example You should create a specific resource group for your users. By using 
exclusive_cpu_cores
, you provide "hard" isolation—the user queries literally cannot touch the cores reserved for the system/ingestion.
Copy code
sql
CREATE RESOURCE GROUP user_queries
TO (role='analyst') -- or map by user
WITH (
    -- CPU: Use 10 out of 16 vCPUs. The remaining 6 are reserved for ingestion.
    "exclusive_cpu_cores" = "10", 
    
    -- Memory: Limit users to 70% of the BE memory (approx 90GB of your 128GB).
    "mem_limit" = "70%",
    
    -- Safety Valve: Kill any single query that exceeds 20GB of RAM on one node.
    "big_query_mem_limit" = "21474836480",
    
    -- Safety Valve: Kill any query using more than 60s of total CPU time.
    "big_query_cpu_second_limit" = "60",
    
    -- Concurrency: Limit to 20 simultaneous queries to prevent thread exhaustion.
    "concurrency_limit" = "20"
);
3. Why this works for Ingestion Health * Ingestion Priority: Ingestion tasks (
INSERT
, 
ROUTINE LOAD
) typically run in the default resource group or a dedicated ingestion group. If you cap the 
user_queries
group to 10 cores, the BE will always have 6 cores available to handle the high-throughput networking and disk I/O required for 
r6in
instances during ingestion. * Spillover Protection: If you notice ingestion is still slow, you can specifically create an ingestion group with 
exclusive_cpu_cores
to "guarantee" those resources. 4. Adjusting for Query Performance To ensure you aren't hurting performance too much: 1. Monitor the "Default" Group: Use 
SELECT * FROM information_schema.resource_group_usage;
to see if your ingestion is actually using those 6 reserved cores. If it's idling, you can increase the user group to 12 cores. 2. Use Query Queues: Instead of failing queries when the limit is hit, enable queuing. This smooths out "bursty" behavior from users without crashing the BE.
Copy code
sql
    SET GLOBAL enable_query_queue_select = true;
    SET GLOBAL query_queue_concurrency_limit = 20;
Summary for 
r6in.4xlarge
 Metric | User Group Setting | Reasoning CPU | 
exclusive_cpu_cores = 10
| Leaves 6 cores for Ingestion and OS. Memory | 
mem_limit = 0.7
| Reserves 30% (~38GB) for loading buffers and metadata. Big Query | 
big_query_mem_limit = 20G
| Prevents a single "rogue" join from OOMing the node. Concurrency | 
concurrency_limit = 20
| Prevents "death by a thousand cuts" (too many small queries). References • best_practices/audit_log_resource_group.md • administration/management/resource_management/resource_group.md
m
@Rocky Does this configuration hurt ingestion? Or the user and ingestion resource group can overlap each other during idle time, and that they just enforce strict boundaries for edge cases
r
This is a great question. The configuration I suggested (using 
exclusive_cpu_cores
) is designed specifically to solve the problem you're describing: it provides strict boundaries for edge cases while still allowing resource overlapping during idle time for ingestion. Here is how the interaction works and why it won't hurt your ingestion health: 1. Hard Limits vs. Resource Borrowing When you set 
exclusive_cpu_cores = 10
for your user group on a 16-core machine: * The Hard Boundary: The user group is "jailed" in those 10 cores. Even if they send a massive query, they cannot touch the remaining 6 cores. This creates a physical "safe zone" for your ingestion and system processes. * The Overlap (Borrowing): By default, StarRocks has a setting called 
enable_resource_group_cpu_borrowing = true
. This means that if your users are idle, the background ingestion tasks (which usually run in the "Shared" pool) can borrow those 10 cores to finish faster. * The Priority: As soon as a user sends a query, the BE engine will immediately reclaim those 10 cores from the ingestion tasks and give them back to the user group. 2. Does it hurt Ingestion? No, it actually protects it. In a "Shared-Nothing" architecture without resource groups, a rogue query would attempt to consume 100% of the CPU on all nodes. This would starve the ingestion threads (RPC handling, memtable flushing, and I/O), leading to "RPC Timeout" or "Table not found" errors during loading. With this configuration: * Ingestion has a guaranteed minimum: It always has at least 6 cores (the "Shared" pool) to handle high-throughput 
r6in
networking and disk flushes. * Ingestion has a higher maximum: It can scale up to all 16 cores if the users aren't doing anything. 3. Understanding Memory (A different story) Unlike CPU, Memory is a hard limit. * If you set 
mem_limit = 70%
for users, they will never be allowed to use more than 70% of the BE's RAM. * This 30% "reserved" memory is vital. Ingestion requires significant memory for Memtables and Loading Buffers. By capping users at 70%, you ensure that a rogue query cannot trigger an Out-Of-Memory (OOM) event that would kill the ingestion process or the BE itself. Recommendation for your 
r6in.4xlarge
 Since 
Routine Load
and 
Stream Load
currently run in the "Shared" pool (not a specific resource group), you should configure your user group as Exclusive.
Copy code
sql
-- This creates a "hard wall" for users but lets Ingestion use the whole machine when idle.
ALTER RESOURCE GROUP user_group WITH (
    "exclusive_cpu_cores" = "10", -- Strict cap for users
    "mem_limit" = "70%",          -- Guaranteed room for loading buffers
    "big_query_mem_limit" = "20G" -- Circuit breaker for rogue queries
);
Summary of Behavior Workload | CPU Usage (Idle Cluster) | CPU Usage (Heavy User Load) User Queries | Up to 10 cores | Exactly 10 cores (Capped) Ingestion | Up to 16 cores (Borrowing) | 6 cores (Guaranteed) This setup achieves exactly what you want: it enforces strict boundaries to protect ingestion from "rogue" behavior, but avoids wasting hardware during the 90% of the time when the cluster isn't under peak-loaded. References • administration/management/resource_management/resource_group.md • loading/RoutineLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md
m
@Rocky So I need to create resource group first, and assign a user to a resource group? How do I know what default user is per application?
r
Yes, the process is rule-based using Classifiers. You don't "assign" a user to a group with a single command; instead, you define a Resource Group and tell it which users/roles/IPs it should automatically "catch." Here is how you can identify your application users and set this up. Step 1: Identify your Application Users If you aren't sure which user your application is using, you can look at the active connections in StarRocks. Run this command while your application is active:
Copy code
sql
-- Shows all current active queries and the users running them
SHOW PROC '/current_queries';
Or for a broader look at all connections:
Copy code
sql
-- Standard MySQL-compatible command to see who is connected
SHOW PROCESSLIST;
* The 
User
column tells you the DB account the app is using. * The 
Host
column helps you verify it's the right app based on its IP address. Step 2: Create the Resource Group with Classifiers Once you have the username (e.g., 
bi_tool_user
), you create the group. The 
TO
clause is the Classifier that maps the user to this group automatically.
Copy code
sql
CREATE RESOURCE GROUP analyst_group
TO 
    (user='bi_tool_user'),           -- Matches a specific user
    (role='reports_role'),           -- Matches anyone with this role
    (source_ip='192.168.1.0/24')      -- Matches any query coming from this subnet
WITH (
    "exclusive_cpu_cores" = "10",
    "mem_limit" = "70%",
    "big_query_mem_limit" = "20G"
);
Step 3: Verify it's working After creating the group, any new query sent by 
bi_tool_user
will automatically be governed by these limits. You can verify this by checking the 
ResourceGroup
column in the query list:
Copy code
sql
SHOW PROC '/current_queries';
Look for your group name in the 
ResourceGroup
column for the queries sent by that user. ────────── Pro-Tips for "Application Identification" 1. Dedicated Users: It is a best practice to create unique users for each application (e.g., 
metabase_user
, 
python_etl_user
). This makes it easy to set different quotas (e.g., your ETL user might need more memory but fewer concurrent slots). 2. IP-Based Rules: If all your users use the same generic account but connect from different servers, you can use the 
source_ip
classifier to separate them into different resource groups. 3. Database-Based Rules: If a "rogue user" is always hitting a specific large "archive" database, you can create a classifier for that database:
Copy code
sql
    ALTER RESOURCE GROUP analyst_group ADD (db='archive_db');
Now, any query hitting `archive_db`—regardless of the user—will be throttled by this group's limits. How to add/change users later If you hire a new analyst or add a new app, you don't need to recreate the group. Just add a new classifier:
Copy code
sql
ALTER RESOURCE GROUP analyst_group ADD (user='new_intern_user');
References • administration/management/resource_management/resource_group.md • using_starrocks/running_queries.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md References • using_starrocks/running_queries.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md
m
@Rocky I run my applications on Ec2 instances. How can I assign each service a user or role? I dont see roles column in processlist
r
In StarRocks, 
SHOW PROCESSLIST
 only shows the User because a StarRocks user is the primary identity that connects to the database. Roles are a collection of permissions granted to that user, and a single user can have multiple active roles at once, which is why there isn't a single "Role" column in the process list. To assign each of your EC2 services its own identity and manage them with Resource Groups, follow this pattern: 1. Create a Dedicated User for Each Service The most reliable way to distinguish between services (e.g., your "API Service" vs. your "Reporting Service") is to give each one its own StarRocks account in its connection string.
Copy code
sql
-- Create a user for your specific EC2 service
CREATE USER 'service_api'@'%';
CREATE USER 'service_reports'@'%';
2. Create Roles for Permission Management Instead of granting permissions directly to users, create Roles. This makes it easy to swap permissions later or apply the same limits to multiple users.
Copy code
sql
-- 1. Create the roles
CREATE ROLE api_role;
CREATE ROLE reports_role;

-- 2. Grant permissions to the roles
GRANT SELECT ON ALL TABLES IN DATABASE prod_db TO ROLE api_role;
GRANT SELECT ON ALL TABLES IN DATABASE archive_db TO ROLE reports_role;

-- 3. Assign the roles to the users
GRANT 'api_role' TO 'service_api'@'%';
GRANT 'reports_role' TO 'service_reports'@'%';

-- 4. CRITICAL: Set these as default so the service doesn't have to run "SET ROLE"
SET DEFAULT ROLE 'api_role' TO 'service_api'@'%';
SET DEFAULT ROLE 'reports_role' TO 'service_reports'@'%';
3. Connect the User/Role to your Resource Group Now you can use the Classifiers we discussed earlier to map these to your resource limits. You can map by user (most common) or by role.
Copy code
sql
-- Map by Role (anyone with this role is limited)
ALTER RESOURCE GROUP user_group ADD (role='reports_role');

-- OR Map by User
ALTER RESOURCE GROUP user_group ADD (user='service_reports');
4. How to see which Role is active Since it's not in 
SHOW PROCESSLIST
, you can check the active role for any specific session by running: * For your current session: 
SELECT current_role();
* To see what roles a user has: 
SHOW GRANTS FOR 'service_reports'@'%';
Why you don't see roles in Processlist In StarRocks' RBAC (Role-Based Access Control) system, a user like 
service_reports
might have 3 different roles: 
db_admin
, 
reports_role
, and 
public
. Since all three can be active simultaneously to give the user the "union" of those permissions, a single column in the process list wouldn't be able to display them clearly. Summary of Workflow for EC2 1. EC2 Side: Update your application's 
.env
or config file to use a unique StarRocks user (e.g., 
user=service_api
). 2. StarRocks Side: Create that user and grant it a specific Role. 3. Resource Group: Create a Classifier for that user or role. 4. Verification: Run 
SHOW PROCESSLIST
to see the 
User
column. If it says 
service_api
, your resource group rules are being applied. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md • sql-reference/sql-statements/user-account-management/SET_DEFAULT_ROLE.md • sql-reference/sql-statements/user-account-management/SHOW_ROLES.md
m
@Rocky So to summarize my understanding, force each application to always define a starrocks role in it's env variable. So during HTTP connection to FE node, we always have an assigned user. Then tie this user to a role and role to resource group?
r
That is a very close summary, but there is one technical nuance to clarify regarding Users vs. Roles during the connection phase. In the StarRocks (and MySQL) protocol, an application authenticates as a User, not a Role. Roles are collections of permissions that are "activated" once a user logs in. Here is the exact, recommended workflow for your EC2 services: 1. The Setup (Admin Side) * Create the User: Create a unique user for the application (e.g., 
app_service_1
). * Create the Role: Create a role that defines what the app can do (e.g., 
api_read_role
). * The Link: Grant the role to the user and set it as the Default.
Copy code
sql
    GRANT 'api_read_role' TO 'app_service_1'@'%';
    SET DEFAULT ROLE 'api_read_role' TO 'app_service_1'@'%';
2. The Application Configuration (EC2 Side) Your application environment variables should define the User, not the Role: * 
STARROCKS_USER=app_service_1
* 
STARROCKS_PASSWORD=your_password
* 
STARROCKS_HOST=your_fe_ip
3. The Resource Group Mapping You can now tie the Resource Group to either the User or the Role. * Matching by User: This is the most direct way if one service = one user.
Copy code
sql
    ALTER RESOURCE GROUP user_group ADD (user='app_service_1');
* Matching by Role: This is better if you have 10 different EC2 services that should all share the same 10-core limit. You just give them all the same role.
Copy code
sql
    ALTER RESOURCE GROUP user_group ADD (role='api_read_role');
4. About the "HTTP Connection" While StarRocks supports a RESTful SQL API (over HTTP), most EC2 applications (Java, Python, Go) use the MySQL protocol (Port 9030) via a standard driver (JDBC/SQLAlchemy/etc). * If using MySQL Protocol: Use the logic above. * If using HTTP SQL API: You still pass the 
user
and 
password
in the Basic Auth header. StarRocks will identify the user and apply the Resource Group rules exactly the same way. Summary Checklist 1. Define a unique User per application/service in your EC2 env vars. 2. Assign a Role to that user and make it the 
DEFAULT ROLE
so it's always active. 3. Add a Classifier to your Resource Group using either the 
user
name or the 
role
name. 4. Verify: Run 
SHOW PROCESSLIST
to see the 
User
column and confirm the application is connecting with the correct identity. This setup ensures that even if one application goes "rogue," its CPU usage is physically jailed to its assigned cores, protecting your ingestion and other services. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/user-account-management/SET_DEFAULT_ROLE.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md
m
@Rocky Is there a way to manage this on terraform? And for the user and password, I can store it in env variables but still need to specify it in my HTTP connection to query the FE node right?
r
Currently, managing StarRocks-specific internal resources (like Resource Groups, Users, and Roles) via Terraform is not yet supported by an official provider. However, you can achieve this using standard workarounds or by leveraging the MySQL-compatible nature of StarRocks. 1. Managing StarRocks with Terraform Since StarRocks uses the MySQL wire protocol, you can use the community-maintained MySQL Provider for Terraform to manage Users and Roles. * Users & Roles: Use the 
mysql_user
and 
mysql_role
resources. * Resource Groups: Because "Resource Group" is a custom StarRocks SQL extension (not standard MySQL), you cannot use the MySQL provider for it. Instead, use a 
null_resource
with a 
local-exec
provisioner to run the SQL command via a MySQL client. Example Terraform snippet:
Copy code
hcl
# Manage the User
resource "mysql_user" "app_service" {
  user               = "service_api"
  host               = "%"
  plaintext_password = var.app_password
}

# Manage the Resource Group (Using local-exec)
resource "null_resource" "starrocks_resource_group" {
  provisioner "local-exec" {
    command = <<EOT
      mysql -h ${var.sr_host} -P 9030 -u root -p${var.root_password} -e \
      "CREATE RESOURCE GROUP IF NOT EXISTS api_group TO (user='service_api') WITH ('exclusive_cpu_cores' = '5');"
    EOT
  }
}
────────── 2. HTTP Connection & Authentication You are correct regarding the HTTP connection. If you are using the HTTP SQL API (introduced in v3.2) to query the FE node: * Authentication: You must provide the user and password in the Basic Auth header. * Role Activation: The HTTP SQL API currently supports a limited set of statements (
SELECT
, 
SHOW
, 
EXPLAIN
, 
KILL
). It may not support the 
SET ROLE
command in the same way a persistent MySQL session does. * The Solution: Use 
SET DEFAULT ROLE
. By assigning a default role to the user, StarRocks automatically activates that role (and its associated Resource Group mapping) the moment the HTTP request authenticates. Example Curl Command for your EC2 Service:
Copy code
bash
curl -X POST 'http://<fe_ip>:8030/api/v1/catalogs/default_catalog/databases/prod_db/sql' \
-u 'service_api:your_password' \
-d '{"query": "SELECT * FROM my_table LIMIT 10;"}' \
--header "Content-Type: application/json"
Key Takeaways for your EC2 Setup 1. Identity: Each EC2 service should have a unique StarRocks User (defined in your service's 
.env
). 2. Permissions: Create a Role, grant it permissions, and 
SET DEFAULT ROLE
for that user. 3. Throttling: Create a Resource Group with a Classifier that matches that specific 
user
or 
role
. 4. Automation: Use Terraform's MySQL provider for users/roles and 
local-exec
for the Resource Group commands. References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/user-account-management/SET_DEFAULT_ROLE.md • using_starrocks/running_queries.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md • faq/resource_isolation_faq.md
m
@Rocky summarize the course of action for me. From resource group creation, role creation, user creation, assigning role to resource group and role to user
r
To set up a complete resource isolation environment for your EC2 services, you should follow this specific order of operations. This ensures that permissions are set before the user connects and that the resource group is ready to "catch" the incoming queries based on the identity you've established. Step 1: Create the Role The role acts as a container for permissions. It's best to create this first so you can assign it to users immediately upon their creation.
Copy code
sql
-- Define the role
CREATE ROLE service_api_role;

-- Grant necessary permissions to the role
GRANT SELECT ON ALL TABLES IN DATABASE prod_db TO ROLE service_api_role;
Step 2: Create the User Create a unique user identity for your EC2 application.
Copy code
sql
-- Create the user (identifies the specific service)
CREATE USER 'ec2_api_service'@'%' IDENTIFIED BY 'your_secure_password';
Step 3: Link Role to User (and Set Default) You must grant the role to the user and—most importantly for HTTP connections—set it as the Default Role. This ensures the role (and its associated resource limits) is automatically active when the app connects.
Copy code
sql
-- 1. Assign the role to the user
GRANT 'service_api_role' TO USER 'ec2_api_service'@'%';

-- 2. Make it active by default
ALTER USER 'ec2_api_service'@'%' DEFAULT ROLE 'service_api_role';
Step 4: Create the Resource Group Define the physical hardware limits (CPU/Memory).
Copy code
sql
CREATE RESOURCE GROUP api_service_group
WITH (
    "exclusive_cpu_cores" = "4", -- Reserve 4 cores for this group
    "mem_limit" = "20%",         -- Limit to 20% of BE memory
    "concurrency_limit" = "50"   -- Max 50 simultaneous queries
);
Step 5: Assign Identity to Resource Group (Classifiers) In StarRocks, you "assign" a role or user to a resource group by creating a Classifier. This tells StarRocks: "If a query comes from this role/user, run it in this specific group." You can do this during creation (Step 4) or add it to an existing group:
Copy code
sql
-- Option A: Map by Role (Recommended if multiple users share the same role)
ALTER RESOURCE GROUP api_service_group ADD (role='service_api_role');

-- Option B: Map by User (Direct mapping)
ALTER RESOURCE GROUP api_service_group ADD (user='ec2_api_service');
────────── Course of Action Summary Table Step | Action | SQL Command | Why? 1 | Create Role | 
CREATE ROLE
| Defines what the service is allowed to do. 2 | Create User | 
CREATE USER
| Provides the login identity for your EC2 
.env
. 3 | Link User/Role | 
SET DEFAULT ROLE
| Ensures the identity is "active" immediately on login. 4 | Create Group | 
CREATE RESOURCE GROUP
| Sets the physical "jail" (CPU/RAM limits). 5 | Link Group | 
ALTER...ADD (role='...')
| Tells the engine to put that role's queries into that group. Verification Once your EC2 application connects, you can verify the setup is working by running:
Copy code
sql
-- See which user is connected (matches Step 2)
SHOW PROCESSLIST;

-- See which resource group is currently being used by active queries
SELECT * FROM information_schema.resource_group_usage;
References • administration/user_privs/authorization/User_privilege.md • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/user-account-management/SET_DEFAULT_ROLE.md References • administration/user_privs/authorization/User_privilege.md • administration/management/resource_management/resource_group.md
m
@Rocky Why not I just define exlucisvely what I must reserve for ingestion and dont bother for user? So query can be however it likes but i know for sure ingestion will always be fine?
r
While your idea of "reserving for ingestion and ignoring users" is logically sound, the implementation in StarRocks has a technical catch based on which type of ingestion you are using. 1. The Technical Limitation Not all ingestion types are currently supported by the Resource Group feature. As of StarRocks v3.3.x: * Supported: 
INSERT INTO ... SELECT
and 
Broker Load
(can be classified using 
query_type = 'insert'
). * Not Supported: 
Routine Load
(Kafka), 
Stream Load
(HTTP), and 
Schema Change
. These run as background/system tasks and do not currently "hit" the resource group classifier logic. 2. How to "Reserve" for Ingestion anyway Because background tasks like Routine Load and Stream Load aren't managed by resource groups, you can't explicitly "put them in a jail." Instead, you do the opposite: you cage the queries. If you want to ensure ingestion always has at least 20% of your CPU, you should: 1. Create a Resource Group for all queries (using a 
role
or the 
default_wg
). 2. Use 
exclusive_cpu_cores
 to hard-limit queries to 80% of your total cores. 3. The remaining 20% of cores are now "untouchable" by the query engine, leaving them available for the background ingestion processes (Routine/Stream Load). Example SQL to "Cage" queries:
Copy code
sql
-- Modify the default group to only allow queries to use 40 cores on a 64-core machine
ALTER RESOURCE GROUP default_wg WITH (
    "exclusive_cpu_cores" = "40" 
);
3. Why the "User + Role" approach is still better If you just limit the "whole system" to leave room for ingestion, you run into two problems: 1. Resource Waste: If no ingestion is happening, those reserved cores sit idle and cannot be used to speed up a big query. 2. The "Rogue Query" Problem: One bad user can still write a query that eats up all the CPU in your "query cage," killing performance for all other users, even if ingestion is safe. Recommended Course of Action (The "Best of Both Worlds") I recommend a hybrid approach: 1. For Ingestion (Routine/Stream Load): Use BE configurations to protect memory. * 
load_process_max_memory_limit_percent
(Default is usually 60%). 2. For Ingestion (Broker/Insert): Create a specific "Ingestion Group" using 
query_type = 'insert'
.
Copy code
sql
    CREATE RESOURCE GROUP ingestion_wg
    TO (query_type in ('insert'))
    WITH ("cpu_weight" = "10", "mem_limit" = "30%");
3. For Users: Stick to the Role-based plan we discussed. This allows you to say: "Service A gets 5 cores, Service B gets 10 cores, and if a data scientist runs a massive join, they only get 2 cores." Summary Table: What to use where Workload | Managed by Resource Group? | Recommendation Routine Load / Stream Load | No | Use BE config for memory; Cage queries for CPU. Broker Load / INSERT INTO | Yes | Use 
query_type = 'insert'
in a Resource Group. App Queries (EC2) | Yes | Use Role-based Resource Groups (as discussed). By defining the user/role, you aren't just "bothering with users"—you are building a multi-tenant safety net. It ensures that "Service A" doesn't starve "Service B," and that both of them together don't starve your critical data ingestion. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • administration/management/BE_configuration.md References • administration/management/resource_management/resource_group.md • loading/RoutineLoad.md
m
@Rocky If I want to map my users 1-1 to resource groups, how should I define the resource groups? I'm thinking the shared memory pool? What about reserved CPUs? If two resource groups belonging to 2 users define 8 reserved CPUs on a 16vCPU machine, does the reserved CPU overlap? I still just want to ensure my ingestion always has a fixed number of CPUs available
r
To achieve 1-1 user mapping while ensuring ingestion is never starved, you need to understand how StarRocks handles Exclusive Cores versus Shared Cores. 1. The CPU Strategy: "Exclusive" vs. "Shared" In StarRocks, CPUs are divided into two distinct buckets: * Exclusive Cores: Assigned to specific resource groups via 
exclusive_cpu_cores
. These cores are "locked" to that group. * Shared Cores: Any CPU core not assigned to an exclusive group. This is where Ingestion (Routine/Stream Load) and the 
default_wg
run. Does the Reserved CPU Overlap? No. 
exclusive_cpu_cores
are additive, not overlapping. * If you have a 16 vCPU machine: * User A: 
exclusive_cpu_cores = 4
* User B: 
exclusive_cpu_cores = 4
* Total Reserved: 8 cores. * Remaining Shared Cores: 8 cores (16 total - 8 reserved). * Constraint: The sum of all 
exclusive_cpu_cores
cannot exceed 
vCPU - 1
. StarRocks forces you to leave at least 1 core for the "Shared" pool. To protect Ingestion: Simply ensure your total 
exclusive_cpu_cores
for all users leaves enough "Shared Cores" for your loading tasks. If your ingestion needs 4 cores to stay healthy, and you have 16 total, only allocate a maximum of 12 cores to your user groups. 2. The Memory Strategy: 
mem_pool
 If you want to map users 1-1 but manage their memory globally, you have two choices: * Individual Isolation: Give each group a 
mem_limit = '10%'
. If User A hits 10%, their query fails, even if User B is idle. * Shared Memory Pool (v4.0+): Put all users into one 
mem_pool
.
Copy code
sql
    -- Both users share a 40% bucket. If User B is idle, User A can use all 40%.
    -- But together, they can never exceed 40%.
    CREATE RESOURCE GROUP user_a_rg WITH ("mem_limit" = "40%", "mem_pool" = "query_tier");
    CREATE RESOURCE GROUP user_b_rg WITH ("mem_limit" = "40%", "mem_pool" = "query_tier");
Recommended Configuration Example On a 16 vCPU machine, to ensure 4 cores are always available for ingestion:
Copy code
sql
-- 1. Create Group for User A (Gets 4 cores exclusively)
CREATE RESOURCE GROUP rg_user_a
TO (user='user_a')
WITH (
    "exclusive_cpu_cores" = "4", 
    "mem_limit" = "20%"
);

-- 2. Create Group for User B (Gets 4 cores exclusively)
CREATE RESOURCE GROUP rg_user_b
TO (user='user_b')
WITH (
    "exclusive_cpu_cores" = "4",
    "mem_limit" = "20%"
);

-- RESULT:
-- User A: Has 4 cores.
-- User B: Has 4 cores.
-- Ingestion/System: Has 8 cores (the "Shared" pool). 
-- NOTE: Ingestion will NEVER be touched by User A or User B's queries.
Important "Borrowing" Note By default, if User A is idle, StarRocks allows "Shared" tasks (like ingestion) to borrow User A's 4 cores. This is efficient. However, if you want to be extremely strict and ensure User A's cores stay cold when they aren't querying, you can disable borrowing:
Copy code
sql
UPDATE information_schema.be_configs 
SET VALUE = "false" 
WHERE NAME = "enable_resource_group_cpu_borrowing";
Summary of Action for You 1. Calculate Ingestion Needs: Determine how many cores your Stream/Routine loads need (e.g., 4 cores). 2. Define User Groups: Create your 1-1 groups using 
exclusive_cpu_cores
. 3. Check the Sum: Ensure 
Sum(exclusive_cpu_cores) <= Total_vCPUs - Ingestion_Cores
. 4. Memory: Use 
mem_limit
per group to prevent a single user from triggering an OOM on the whole BE. References • faq/resource_isolation_faq.md
m
@Rocky So if I have many different services that need to be "rate-limited" per se on queries to not impact ingestion, I eventually run out of cores? And can you re-iterate why I can't do the inverse of giving ingestion exclusive cores?
r
Yes, you have hit on the exact trade-off of "Hard Isolation." If you give every service exclusive cores, you will indeed run out of cores very quickly because those cores are physically partitioned away from the general pool. Here is the breakdown of why the "inverse" doesn't work and how to handle "many services" without running out of resources. 1. Why can't I give Ingestion Exclusive Cores? The technical reason is Identity & Architecture: * No "Session" Identity: Resource groups work via Classifiers (Who is the user? What is the role?). Routine Loads and Stream Loads are background system tasks managed by internal FE/BE threads. They don't have a "User Session" or a "Role" that the resource manager can catch and put into a group. * The "Shared Pool" Design: StarRocks divides the CPU into two buckets: the Exclusive Pool and the Shared Pool. * Exclusive Groups take cores out of the system. * Shared Groups (and all Ingestion) live in the "Shared Pool." * System Protection: Because ingestion is the lifeblood of the database, it is hard-coded to run in the Shared Pool to ensure it always has access to at least 1 core (since you can't assign all 
n
cores to exclusive groups; the limit is 
n-1
). 2. The Solution for "Many Services": Soft Isolation (
cpu_weight
) If you have 20 services, do not use 
exclusive_cpu_cores
. Use 
cpu_weight
instead. How it works: Instead of saying "Service A gets 2 physical cores," you say "Service A has a weight of 10." * If Service A is the only one running, it can use all available cores in the Shared Pool. * If Service B (weight 10) starts running at the same time, the system will split the CPU 50/50. * Crucially: You can have 100 services all using weights, and you will never "run out of cores." They will simply throttle each other proportionally based on their weight. 3. How to ensure Ingestion is fine (The "Inverse" Logic) Since you can't put Ingestion in a cage, you create a cage for everything else. Step 1: Create a "Global User Pool" with a Weight Instead of 1-1 mapping for every single user, group your "regular" services into a shared resource group.
Copy code
sql
CREATE RESOURCE GROUP general_service_rg
WITH (
    "cpu_weight" = "10",  -- High weight for query performance
    "mem_limit" = "60%"   -- Leave 40% for system/ingestion
);
-- Map all your service roles to this one group
ALTER RESOURCE GROUP general_service_rg ADD (role='service_role_1');
ALTER RESOURCE GROUP general_service_rg ADD (role='service_role_2');
Step 2: Use the "CPU Buffer" Trick If you want to be 100% sure Ingestion always has 4 cores available on a 16-core machine: 1. Do not give your services 
exclusive_cpu_cores
. 2. Create one "Dummy" or "Administrative" group with 
exclusive_cpu_cores = 11
. 3. Assign your users to a Shared Group (using 
cpu_weight
). 4. Because the system now only has 5 cores left in the "Shared Pool" (16 total - 11 exclusive), your weighted services and your Ingestion will compete for those 5 cores. Summary: The "Best Practice" for Many Services If you have a high number of services: 1. Don't use Exclusive Cores for the services. It is too rigid. 2. Use 
cpu_weight
 to allow services to share the CPU. 3. Protect Ingestion via Memory: Use 
mem_limit
on your resource groups to ensure queries don't trigger a global Out-of-Memory (OOM) that kills the ingestion threads. 4. Protect Ingestion via `concurrency_limit`: Limit how many queries can run at once. This prevents "CPU death by a thousand cuts" where too many small queries starve the background ingestion. Rule of Thumb: * Exclusive Cores: Use only for your #1 most critical, latency-sensitive app (e.g., your Executive Dashboard). * CPU Weight: Use for everything else (including your "many services"). This/That services"). This" services). This" services). References
m
@Rocky I dont mean assignign stream load a resource group. I know I have a certain application performing the stream load. Can I not give it the resource group to guarantee ingestion resources?
r
The short answer is no, you currently cannot assign a Resource Group to a Stream Load, even if you know the specific application/user performing it. Here is the technical "Why" and the workaround to achieve exactly what you want. 1. Why Stream Load ignores Resource Groups Resource Groups in StarRocks are designed to manage the Pipeline Execution Engine. This engine handles the complex logic of 
SELECT
queries and 
INSERT INTO ... SELECT
/ 
Broker Load
jobs (which are essentially big distributed queries). Stream Load is different: * Lightweight Path: It is an HTTP-based write directly to the storage engine (MemTable). It uses dedicated thread pools on the Backend (BE) rather than the general Pipeline threads. * No Classifier Check: The BE code that handles the incoming Stream Load data stream doesn't "check-in" with the Resource Manager to see which CPU bucket it belongs to. It simply runs in the system's Shared Pool. 2. How to "Guarantee" Ingestion Resources (The Workaround) Since you cannot put Stream Load into a "Reserved Cage," you must cage everything else. Think of your CPU as two separate buckets: 1. The Exclusive Pool: Cores you explicitly give to Resource Groups. 2. The Shared Pool: Cores left over. All Stream Load and Routine Load run here. If you have a 16 vCPU machine and you want to ensure ingestion always has 4 cores available: Step A: Fence off the Querying Users You create a resource group for your query applications and give them a hard "Exclusive" limit.
Copy code
sql
-- This group will "steal" 12 cores away from the system.
-- Queries in this group CANNOT use the remaining 4 cores.
CREATE RESOURCE GROUP query_app_rg
TO (role='my_query_role')
WITH (
    "exclusive_cpu_cores" = "12",
    "mem_limit" = "60%"
);
Step B: The Result * Query Apps: Limited to 12 cores. They can never spike to use the full 16. * Ingestion: Because it has no resource group, it defaults to the Shared Pool. * The Shared Pool: Has exactly 4 cores left (16 total - 12 exclusive). Because your queries are physically forbidden from touching those 4 cores, Ingestion now has a 4-core guarantee. 3. Other "Knobs" for Stream Load Since you are looking to protect ingestion specifically for an application, you should also look at these BE Configurations (in 
be.conf
): * 
load_process_max_memory_limit_percent
 (Default 30%): This is the global "safety valve" for all ingestion. If your Stream Load application is huge, you might want to increase this to ensure it doesn't fail when memory is tight. * 
push_worker_count_high_priority
: Controls the number of threads dedicated to handling ingestion tasks. * Merge Commit (v3.4+): If your application sends many small Stream Loads, enable Merge Commit. This combines many small loads into one transaction, which drastically reduces the CPU/IO spent on "Compaction" (the background cleanup that usually slows down a busy system). Summary You can't give Ingestion a Resource Group because it doesn't "live" in the Resource Group world. But you can guarantee its resources by using 
exclusive_cpu_cores
to restrict your query users, effectively leaving a "hole" in the CPU capacity that only Ingestion can fill. References • administration/management/resource_management/resource_group.md • loading/StreamLoad.md • administration/management/BE_configuration.md References • loading/Loading_intro.md • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md
m
@Rocky But that means ingestion can't access shared pool when needed? My CPU utilization on average slightly below 50 across all nodes but occasionally spikes to slightly below 70. How should I architecture this?
r
To architect this correctly, you need to choose between Hard Isolation (Exclusive Cores) and Soft Isolation (CPU Weight). Based on your observation that your CPU utilization spikes but averages 50%, here is the best way to handle the 1-1 mapping while protecting ingestion. The "Exclusive" Architecture (Hard Isolation) If you define your 1-1 users with 
exclusive_cpu_cores
, you are essentially carving the machine into pieces. * How it protects Ingestion: You don't give ingestion a group; instead, you give everyone else a group. The cores you don't assign to users automatically become the Shared Pool. This is where Ingestion (Stream Load) and the 
default_wg
live. * The "Bursting" behavior: By default, the configuration 
enable_resource_group_cpu_borrowing
is set to 
true
. This means: * Ingestion CAN burst: If ingestion needs more than the Shared Pool cores, it can borrow idle cores from the users' exclusive groups. * Users CANNOT burst: Users are strictly capped. Even if ingestion is totally idle, a user with 2 exclusive cores will never use 3. Verdict: Use this if your primary goal is to ensure ingestion is never "starved" by a massive query spike from a user. The "Weighted" Architecture (Soft Isolation) If you want both users and ingestion to be able to use 100% of the CPU when the machine is idle, you should use 
cpu_weight
. * How it works: You don't reserve cores. Instead, you give each user a "Weight" (e.g., 
cpu_weight = 1
). * The 
default_wg
priority: The 
default_wg
(where ingestion and system tasks live) has a default weight equal to the total number of vCPUs on your BE. * Protection: If you have 10 users with 
weight = 1
and a 16-core machine, the total weight is 26 ($10 + 16$). * Ingestion/System gets $16/26$ (~61%) of the CPU when everyone is fighting for resources. * Users get $1/26$ (~4%) each. * Bursting: If only one user is active, they can take all 16 cores. If only ingestion is active, it takes all 16 cores. Verdict: Use this if you want maximum efficiency and are okay with users occasionally competing with ingestion during the rare moments when everyone spikes at once. Addressing your specific concerns 1. "Does that mean ingestion can't access shared pool?" Actually, ingestion is the Shared Pool. In StarRocks, anything not in an exclusive group is "Shared." The concern is usually the opposite: ensuring queries don't eat the shared pool. 2. "Why can't I give ingestion exclusive cores?" Because Stream Load is an internal system task. It doesn't use the standard SQL execution pipeline that the Resource Manager monitors. It uses dedicated background thread pools. Since it doesn't have a "User" or "Query ID" in the same way a SELECT query does, the classifier can't catch it and put it in a cage. 3. Recommendations for your 70% spikes: Since you have many services and occasional spikes, Soft Isolation (
cpu_weight
) is typically better. It prevents you from "running out" of cores and allows your idle capacity to be used by whichever service needs it most at that millisecond. Summary Checklist * For 1-1 mapping + Bursting: Use 
CREATE RESOURCE GROUP ... WITH ("cpu_weight" = "1")
. * To protect Ingestion in this setup: Set a 
mem_limit
(e.g., 5-10%) on each user group. This ensures no single user query can trigger an Out-of-Memory event that kills the ingestion threads. * To monitor: Use 
SELECT * FROM information_schema.resource_group_usage
to see which users are hitting their limits during your 70% spikes. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md
m
@Rocky Elaborate on the soft isolation architecture. Why is total weight 16 when we have 10 users each with weight 1 on a 16 core machine? dont each user get 10%?
r
This is the most common point of confusion with "Soft Isolation" in StarRocks. The reason the math doesn't result in "10% per user" is because of the System Default Group, which acts as the "Protective Shell" for your ingestion and system tasks. Here is the breakdown of the calculation for a 16-core machine: 1. The Invisible Participant: 
default_wg
 In StarRocks, there is a hidden resource group called 
default_wg
. This is where: * Ingestion (Stream Load, Routine Load) runs. * System tasks (Compaction, Metadata sync) run. * Any user not explicitly assigned to a resource group runs. By default, the 
cpu_weight
of 
default_wg
is equal to the number of vCPUs on the BE. * On your 16-core machine, 
default_wg
has a weight of 16. 2. The Calculation (Total Weight = 26) When you create 10 resource groups (one for each service/user) and give each a weight of 1, the scheduler looks at the total "tickets" in the system: * 10 Users/Groups: $10 \times 1 = 10$ tickets * default_wg: $16$ tickets (Automatic) * TOTAL: 26 tickets 3. Why users don't get 10% In a period of full contention (when every service and ingestion are all trying to use 100% CPU at the exact same time), the resources are split like this: Group | Weight | Percentage Share | Core Equivalent System (Ingestion) | 16 | $16 / 26 \approx \mathbf{61.5\%}$ | ~9.8 Cores User 1 | 1 | $1 / 26 \approx \mathbf{3.8\%}$ | ~0.6 Cores User 2 | 1 | $1 / 26 \approx \mathbf{3.8\%}$ | ~0.6 Cores ... | ... | ... | ... User 10 | 1 | $1 / 26 \approx \mathbf{3.8\%}$ | ~0.6 Cores This is actually a safety feature. It ensures that even if all 10 of your services go crazy with queries, the System/Ingestion group still retains the majority (~61%) of the CPU power because its weight (16) is much higher than the individual user weights (1). 4. How to get the "10% per user" you want If you want your users to have more "clout" relative to ingestion, you have two choices: Option A: Lower the System Weight You can manually lower the weight of the system group so it doesn't "hog" so many tickets.
Copy code
sql
ALTER RESOURCE GROUP default_wg WITH ("cpu_weight" = "2");
* Now: 10 (Users) + 2 (System) = 12 total tickets. * Each User: $1/12 \approx 8.3\%$ * System: $2/12 \approx 16\%$ Option B: Increase User Weights Give each user a weight equal to the core count if you want them to be "equal" to the system.
Copy code
sql
-- Do this for each of your 10 groups
ALTER RESOURCE GROUP user_service_1 WITH ("cpu_weight" = "16");
* Now: $(10 \times 16) + 16 = 176$ total tickets. * Each User: $16/176 \approx 9.1\%$ (Very close to your 10% goal). Why this is better than "Hard" cores: In your scenario (50% average load, 70% spikes), if a user has a weight of 1, they are only "throttled" to 3.8% if, and only if, the machine is at 100% capacity. If the machine is at 50% capacity, that user can "burst" and use all 50% of the remaining idle CPU. The weight only matters when there is a line at the door. This flexibility is why Soft Isolation is the recommended architecture for the "many services" use case. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • best_practices/audit_log_resource_group.md
m
@Rocky So with soft isolation you would recommend letting default_wg stay at number of cores and each user just gets 1 weight? Let's say 1 user actively querying. So it gets max 1/17 (5.9%) of CPU. However, what ingestion doesn't actually need the remaining 94.1% of cpu utilization since they average around 50% CPU util. Does this mean since most of the machine would then be idle on the ingestion side, the user can borrow more cpus?
r
In your specific architecture (many services + variable spikes), yes, your active users can absolutely "borrow" CPU when the ingestion side is idle. Here is how the "Soft Isolation" logic handles your 50%-70% utilization scenario: 1. The "100% Load" Fallacy The "5.9% per user" math ($1 / 17$) is a Worst Case Scenario. It only applies if the server is pinned at 100% CPU and every single group is trying to consume as much as possible at the same millisecond. In your case, where you average 50% utilization: * The Scheduler is smart: If 
default_wg
(ingestion/system) is only using 30% of its "60% share," the remaining 30% isn't wasted. The StarRocks scheduler immediately offers those idle cycles to any other active group. * Single User Burst: If only one user is querying and ingestion is at its 50% baseline, that single user can easily burst to consume all remaining CPU capacity (the other 50%). 2. Why 
default_wg
having 16 Weight is your "Safety Buffer" Think of the 16 tickets assigned to 
default_wg
as a Priority Reserve rather than a hard limit. * When Ingestion Spikes: If a massive Stream Load hits and the system utilization jumps toward 90%, the 16 weight of 
default_wg
ensures it can "push back" against the query users. It says: "I have 16 tickets, you only have 1; I get to use the CPU first." * When Ingestion is Quiet: When ingestion is doing nothing, those 16 tickets are effectively "off the table." The total "tickets" in the system drops from 26 down to 10 (your users). Now each user's weight of 1 represents $1/10$ (10%) of the total available power. 3. The "Unmanaged" Ingestion Factor (Crucial Detail) As established, Stream Load is not currently managed by the Resource Group scheduler (it doesn't appear in the 
cpu_weight
table). It runs in background thread pools (
push_worker
). Because Stream Load is "invisible" to the resource groups: 1. Ingestion is the "Noisy Neighbor": It will take the CPU it needs directly from the OS. 2. Resource Groups are the "Self-Governing Tenants": Your 10 users share whatever is left over after the OS handles ingestion. 3. The Result: By giving users a 
cpu_weight
of 1, you are telling the StarRocks query engine: "When we are fighting for the CPU that ingestion hasn't taken, split it evenly among these 10 services." Architecture Recommendation For your specific 70% spike environment, the Soft Isolation (Weights) approach is better than Hard Isolation (Exclusive Cores) for the following reason: If you used Exclusive Cores (Hard Isolation) to reserve 4 cores for ingestion, those 4 cores would be legally forbidden from being used by queries. Even if ingestion was 0% utilized and a user was waiting on a huge query, those 4 cores would sit idle. Stick with Soft Isolation: * Set 
default_wg
weight to your core count (16). * Set each User group weight to 1. * The Benefit: Your users can burst to use 100% of the machine at midnight when ingestion is idle, but ingestion is protected by the "Priority Reserve" when things get busy during the day. Summary of the Math for your 50% Load: * Total Capacity: 16 Cores * Ingestion (at 50%): Consumes 8 Cores * Remaining: 8 Cores * Active User 1: Can use all 8 remaining cores ($100\%$ of the available slack). * If Users 1 and 2 query together: They will each get 4 cores ($1/2$ of the available slack). References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • best_practices/audit_log_resource_group.md References • best_practices/audit_log_resource_group.md
m
@Rocky But as my number of users grow, let's say to 20, I would then need to adjust the default_wg right? Because then it's share drops from 16/26 (61.5%) to 16/36 (44.%), so I always need to tune these the values.
r
You are exactly right. As you add more resource groups, the "weight pool" grows, and the relative share of the 
default_wg
(and thus your ingestion protection) shrinks. If you add 20 users each with a weight of 1, the 
default_wg
(weight 16) now only commands ~44% of the CPU during a "traffic jam." To avoid the "constant tuning" trap as your service grows, here are the three best ways to architecture this: 1. The "High-Floor" Strategy (Recommended) Instead of leaving 
default_wg
at the default (16), set it to a much higher number that makes individual user additions "noise" in the math. * Action: 
ALTER RESOURCE GROUP default_wg WITH ("cpu_weight" = "100");
* With 10 users (weight 1): Ingestion share = $100 / 110 = \mathbf{91\%}$ * With 50 users (weight 1): Ingestion share = $100 / 150 = \mathbf{66\%}$ * Result: You don't have to touch the configuration again until you have dozens of new services. This ensures that the system/ingestion always has the "biggest stick" in the room. 2. The "Tiered" Architecture Instead of 1-1 mapping for every service, group your services into Tiers (Gold, Silver, Bronze). * Create 3 resource groups: 
prod_high
, 
prod_med
, 
prod_low
. * Assign your 20 users to these 3 groups using classifiers. * Why this scales: The total number of resource groups stays at 3, regardless of whether you have 20 users or 200 users. Your 
default_wg
share remains perfectly stable. 3. Re-evaluating "Hard" vs "Soft" Isolation for Scaling If your goal is to never tune the ingestion protection, the Hard Isolation (Exclusive Cores) for users is actually more "set and forget." * Architecture: * Total cores: 16. * Create 10 User Groups with 
exclusive_cpu_cores = 1
. * Result: 10 cores are reserved for users. The remaining 6 cores are the Shared Pool (where ingestion lives). * Scaling to 20 Users: You literally cannot. The system will stop you because $20 > 16$. * The Trade-off: Hard isolation forces you to think about capacity planning before you add a service, whereas Soft isolation (Weights) lets you add infinite services but dilutes the power of each one. Rocky's Summary Recommendation For a growing environment with many services: 1. Use Soft Isolation (
cpu_weight
) to allow bursting. 2. Boost the 
default_wg
weight to 100 immediately. This gives you a massive safety buffer so that your ingestion remains the highest priority even as you scale from 10 to 50 services. 3. Monitor with this query:
Copy code
sql
    -- Check the current relative percentages of all groups
    SELECT name, cpu_weight,
           (cpu_weight / SUM(cpu_weight) OVER()) * 100 as potential_cpu_percentage
    FROM information_schema.resource_group_usage;
This way, you only need to tune the values once or twice a year rather than every time a new service is onboard. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md References • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md
m
@Rocky So going to the extreme and making default_wg 100 gives it the biggest stick in the room. A new user will only get ~1% of CPU at worst. However, since CPU is 50% idle most of the time, it should never happen and queries should always work. So this is the easiest safeguard for ingestion? And we can balance the remaining users to 1 so we never worry about queries contending with ingestion
r
Your logic is mostly correct, but there is one critical technical detail regarding the 
cpu_weight
limit that changes how you should implement this "big stick" strategy. 1. The Maximum Weight Limit In StarRocks, the 
cpu_weight
parameter has a specific constraint: its value range is (0, number of CPU cores]. * On your 16-core machine, the maximum weight you can assign to any group (including 
default_wg
) is 16. * You cannot set it to 100. If you try, the system will likely throw a validation error. 2. How to achieve the "1% Worst-Case" (The Reverse Strategy) Since you can't push 
default_wg
up to 100, you achieve the exact same "safeguard" by pushing the user weights down. Instead of 10 users with Weight 1 and a System with Weight 16, do this: * System (
default_wg
): Keep at 16 (its max). * Users/Services: Set each to 0.1 (or the smallest allowed increment, typically 1 or a small decimal depending on your StarRocks version). The Math at 100% Contention (with 20 users at weight 0.1): * Total weight = 16 (System) + 2 (20 users × 0.1) = 18. * System Share: $16 / 18 = \mathbf{88.8\%}$ * User Share: $0.1 / 18 = \mathbf{0.5\%}$ This effectively gives ingestion the "Big Stick" you were looking for. 3. Will queries still work? (The "Borrowing" Reality) Yes. This is the beauty of the Soft Isolation architecture. Your assessment of the 50% idle capacity is spot on: * During Normal Operation (50% Ingestion Load): Even if a user only has a "guaranteed" share of 0.5%, they are not restricted to it. Because the scheduler is work-conserving, that user can borrow every single idle cycle. If ingestion is using 8 cores and the other 8 are idle, a single query can "burst" and use all 8 idle cores. * During a Crisis (100% Ingestion Spike): If a massive data load suddenly requires all 16 cores, the scheduler will immediately claw back the "borrowed" cores from the users. The user's query won't necessarily fail, but it will slow down significantly as it is compressed into its 0.5% "reserve" to make room for the ingestion. 4. Is this the easiest safeguard? Yes, it is the safest "hands-off" configuration. By keeping the user weights very low and the 
default_wg
weight at the core count: 1. Ingestion is Protected: You never have to worry about a "rogue" query (like a massive 
SELECT *
without a limit) killing your real-time data pipelines. 2. Resources aren't Wasted: You aren't "locking" cores away. Users get the full power of the 16-core machine whenever ingestion isn't using it. 3. Low Maintenance: As you grow from 10 to 20 to 50 users, you don't need to keep re-tuning. Even at 50 users, the system still prioritizes the 
default_wg
because its weight (16) remains dominant over the sum of many small user weights. Summary Recommendation To implement your plan on a 16-core machine: 1. Verify 
default_wg
is at 16 (it should be by default). 2. Set your user-facing resource groups to a weight of 1 (or smaller if your version allows decimals). 3. If you have 20 users at weight 1, ingestion is guaranteed ~44% ($16/36$). 4. If you want ingestion to have a higher guarantee (like 80%+), you should simply keep the number of resource groups low by grouping users into Tiers (e.g., 
prod_group
, 
test_group
) rather than one group per user. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/ALTER_RESOURCE_GROUP.md References • faq/resource_isolation_faq.md
m
@Rocky What is the maximum granularity of cpu_weight. What happens if ingestion weight is at most 90% of the CPU, and a certain user that is only allocated 1% cpu weight does a bursty query? It will try to borrow up to 50% since it's idle anyways right? What if multiple users are trying to contend beyond their 1% allocated CPU since there'll always be idle CPU in the shared pool?
r
To answer your questions about the precision and behavior of 
cpu_weight
, here is the breakdown of how StarRocks handles these "Soft Isolation" scenarios: 1. Maximum Granularity of 
cpu_weight
 The granularity of 
cpu_weight
is integer-based. * Data Type: It is an 
INT
. You cannot set it to 
0.1
or 
0.5
. The minimum valid value is 1. * Value Range: The range is (0, 
avg_be_cpu_cores
]. This means on your 16-core machine, the maximum weight you can assign to any single group (including 
default_wg
) is 16. The Impact on your "1% Target": Because the minimum weight is 1 and the maximum is 16, you cannot technically achieve a "1% guarantee" if you have 20 individual user groups. * With 20 users (weight 1 each) and 
default_wg
(weight 16): * Total weight = 36. * Each user's worst-case share is $1/36 \approx \mathbf{2.8\%}$. * Ingestion's worst-case share is $16/36 \approx \mathbf{44.4\%}$. 2. The "Bursty User" vs. Idle Ingestion If Ingestion is allocated 90% via weights but is only using 0% (idle), and a User with a 1% weight starts a massive query: * Borrowing Logic: The StarRocks scheduler is work-conserving. It does not care that the user is "only allocated 1%." If the other 99% of the "weight tickets" aren't being used by anyone else, the active user can borrow up to 100% of the CPU. * Response to Ingestion: The moment an ingestion task arrives, the scheduler will immediately (within milliseconds) reclaim those cycles. The user's query will be squeezed back down toward its 2.8% share to allow the ingestion to take its 44.4% (or more, if other users are idle). 3. Multiple Users Contending for Idle CPU This is where the "Shared Pool" logic becomes very fair. Imagine Ingestion is idle, but User A and User B (both weight 1) both start heavy queries: 1. Redistribution of Idle Capacity: The 16 weight "tickets" from the idle 
default_wg
are effectively ignored by the scheduler. 2. Relative Weighting: The scheduler looks at the active groups. Since User A has 1 ticket and User B has 1 ticket, they share the total CPU 50/50. 3. The Formula: 
Your Share = (Your Weight) / (Sum of weights of all ACTIVE groups)
. 4. Scaling to 10 Users: If 10 users are active and ingestion is idle, each user gets $1/10$ (10%) of the total CPU. Rocky's Architecture Tip for "Infinite" Scaling If you want to ensure Ingestion always has a "Big Stick" (e.g., >80% protection) regardless of how many users you add, you should stop creating 1 resource group per user. Instead: * Group users into a "Query Pool": Create one resource group called 
user_queries
with 
cpu_weight = 2
. * Map all 20+ users to this one group using classifiers. * The Result: * 
default_wg
(Ingestion) weight = 16. * 
user_queries
group weight = 2. * Ingestion is now guaranteed $16/18$ (88.9%) of the CPU forever, no matter if you have 20, 100, or 500 users. _ The 500 users will simply fight amongst themselves _within* that 2-weight share during high load, but they will never be able to collectively "drown out" the ingestion. Summary * Max Granularity: Integer (1 is the minimum). * Max Weight: Core count (16). * Single User Burst: Can take 100% if everyone else is idle. * Multi-User Contention: Idle CPU is shared proportionally based on the weights of the groups currently running queries. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md References
Open in Slack
PreviousNext
Powered by