@Rocky any recommendations for setting up resource based queues for the following access patterns interactive/realtime, ingestion (INSERT/DELETE), APIs/Data Exports and processes that can take up to 10mins on a single m8gd.2xlarge CN using Shared Data mode.

For a single m8gd.2xlarge (8 vCPUs, 32 GB RAM) CN node in Shared Data mode, you need to balance responsiveness for interactive queries against the heavy load of ingestion and 10-minute long-running processes. Since you only have 8 cores, strict isolation is key to preventing a single long-running export or a heavy ingestion task from starving your real-time APIs. Here is a recommended configuration using Resource Groups. 1. Interactive & Real-time (High Priority) This group is for low-latency queries. We give it the highest

cpu_weight

and a dedicated memory slice to ensure speed.

sql
CREATE RESOURCE GROUP rg_interactive
TO (role='interactive_role') -- or specific users
WITH (
    'cpu_weight' = '8',            -- High priority relative to others
    'mem_limit' = '0.3',           -- 30% of total memory (~9.6GB)
    'concurrency_limit' = '50'     -- Allow high concurrency for small queries
);

2. Ingestion (INSERT/DELETE) Ingestion is critical but can usually tolerate slightly more latency than a dashboard query. We cap the memory to prevent OOMs during heavy write bursts.

sql
CREATE RESOURCE GROUP rg_ingestion
TO (query_type IN ('insert', 'delete'))
WITH (
    'cpu_weight' = '2',            -- Lower priority than interactive
    'mem_limit' = '0.2',           -- 20% of memory (~6.4GB)
    'concurrency_limit' = '10'     -- Prevents too many parallel write transactions
);

3. APIs & Data Exports These are often "medium-weight" queries. We limit concurrency here because a single large export can easily saturate the 8-core CPU or the network.

sql
CREATE RESOURCE GROUP rg_api_export
TO (user='api_service_user')
WITH (
    'cpu_weight' = '4',            -- Medium priority
    'mem_limit' = '0.2',           -- 20% of memory (~6.4GB)
    'concurrency_limit' = '5',     -- Strict limit to prevent bulk exports from piling up
    'spill_mem_limit_threshold' = '0.7' -- Enable intermediate spilling to disk if memory is tight
);

4. Long-running Processes (Up to 10 mins) For processes taking up to 10 minutes, you must prevent them from occupying all 8 cores simultaneously. We use

big_query_cpu_second_limit

to allow them to run long, but keep

concurrency_limit

very low.

sql
CREATE RESOURCE GROUP rg_long_tasks
TO (role='batch_process_role')
WITH (
    'cpu_weight' = '1',            -- Lowest priority; gets "leftover" CPU
    'mem_limit' = '0.3',           -- 30% of memory (~9.6GB)
    'concurrency_limit' = '2',     -- ONLY allow 1 or 2 at a time on an 8-core node
    'big_query_cpu_second_limit' = '1200' -- 20 minute limit (600s is 10 min, 1200s adds safety)
);

Key Recommendations for your

m8gd.2xlarge

setup: 1. CPU Scheduling: StarRocks uses

cpu_weight

for relative scheduling. If

rg_interactive

and

rg_long_tasks

are running at the same time, the interactive group will get 8/9ths of the CPU cycles (8 weight vs 1 weight). 2. Memory Management: On a 32GB node, roughly 28GB is available for the StarRocks process. Totaling your

mem_limit

percentages to 1.0 (100%) is fine, as StarRocks handles the internal allocation. 3. Spilling: For your APIs/Data Exports, I added

'spill_mem_limit_threshold' = '0.7'

. This is crucial for an 8-core/32GB node; if an export query exceeds 70% of its allocated group memory, it will spill to the local NVMe disk (provided by the

m8gd

instance) instead of failing with an "Out of Memory" error. 4. Monitoring: Use the following command to see how your groups are performing:

sql
    SELECT * FROM information_schema.resource_group_usage;

5. Query Queuing: If you want queries to wait instead of failing when the

concurrency_limit

is hit, ensure global queuing is enabled in your FE config (

enable_query_queue = true

). References

@Rocky How do these numbers change with scaling out CN. Lets say we we added another CN with same spec (2 in total)

When you scale out from 1 Compute Node (CN) to 2 Compute Nodes (with the same specifications), you can expect the following changes in your performance metrics: *1. Query Throughput (QPS)* * Scaling: Nearly Linear. * Behavior: For a high-concurrency workload (many small/medium queries), adding a second CN roughly doubles the cluster's aggregate CPU and memory resources. Since the FE (Frontend) distributes these queries across all available CNs, you should see the total system throughput (Queries Per Second) nearly double, provided the FE itself is not a bottleneck. *2. Query Latency* * Scaling: Sub-linear improvement (varies by query type). * Behavior: * Complex/Large Queries: For heavy queries that involve large scans or joins, StarRocks will distribute the execution fragments across both CNs. If the query was previously compute-bound, latency may decrease significantly (though not necessarily by half) due to the increased parallelism. * Simple Queries: For very "light" queries, you might not see a major change in latency because the overhead of network communication and coordination between two nodes can offset the gains in compute speed. *3. Cache Capacity & Hit Rate* * Scaling: Doubles Total Cache Size. * Behavior: In a shared-data architecture, each CN maintains its own local disk cache for "hot" data. * Benefit: Adding a second CN doubles the total available cache capacity for the cluster. This allows a larger "hot" dataset to fit in the cache, reducing the need to fetch data from remote storage (like S3/HDFS). * Consistency: StarRocks uses data-affinity scheduling to try and route queries for the same data to the same CN, ensuring that the cache is used efficiently even as you scale out. *4. Scaling Overhead* * Behavior: Near Zero. * Advantage: Unlike a shared-nothing architecture (where adding nodes requires "rebalancing" or moving data tablets), scaling CNs is instant. Since CNs are stateless, the new node can start processing queries and building its local cache immediately after it is added to the cluster. Summary Table: 1 CN vs. 2 CNs Metric | 1 CN | 2 CNs Compute Resources | 1x (CPU/RAM) | 2x (CPU/RAM) Max Throughput | Baseline | ~1.8x to 2x Latency (Heavy Query) | Baseline | Reduced (e.g., 30-40% faster) Local Cache Limit | Baseline | 2x Aggregate Cache Data Migration | N/A | None (Immediate effect) Note: To ensure the 2nd CN is fully utilized, make sure your queries are distributed. If you are using a load balancer or a list of IPs in your application, ensure it points to the FE, which will then handle the internal distribution of work to both CNs. References • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md • quick_start/shared-data.md

@Rocky but do the resource group configuration remain the same? For example concurrency limits? Should they double? What remains and what needs scaling with the CNs?

When you scale your Compute Nodes (CNs) in StarRocks, the resource group configurations generally remain the same in terms of their definition, but you often need to manually adjust specific parameters—like concurrency—to take advantage of the additional hardware. Here is a breakdown of what stays the same and what you should scale with your CNs: 1. Concurrency Limits (

concurrency_limit

) Should they change? Yes (Increase them). The

concurrency_limit

in a resource group is an integer value that defines the maximum number of concurrent queries allowed for that group. * Why: In StarRocks, most queries run across all available nodes. If you have a limit of

10

and you double your nodes from 4 to 8, the system is now twice as powerful, but the resource group will still only allow

10

concurrent queries to start. * Action: If your goal for scaling is to support more simultaneous users, you should increase (e.g., double) the

concurrency_limit

. If you don't, the extra nodes will make individual queries faster, but you won't be able to run more of them at the same time. 2. Resource Quotas (

cpu_weight

and

mem_limit

) Should they change? Usually No. These parameters are relative or percentage-based, so they "auto-scale" in terms of absolute resources as you add nodes: * `mem_limit`: This is a percentage (e.g.,

0.5

for 50%). If you add more nodes, the resource group still gets 50% of each node. Because you have more nodes, the total cluster memory available to that group increases automatically. * `cpu_weight`: This is a relative priority (e.g., Group A has weight 10, Group B has weight 20). This ratio remains valid regardless of how many nodes you have. 3. CPU Core Limits (

max_cpu_cores

and

exclusive_cpu_cores

) Should they change? Depends on node specs. * `max_cpu_cores`: This limit is applied per BE/CN node. If you set this to

8

, every node in the cluster (including new ones) will limit that resource group to 8 cores. * Action: You only need to change this if your new CN nodes have a different number of cores than your old ones, or if you want to change the "size" of a query "slot" on a per-node basis. 4. Big Query Limits (

big_query_scan_rows_limit

, etc.) Should they change? Usually No. These are "guardrail" limits designed to kill runaway queries. * `big_query_scan_rows_limit`: This is the maximum rows a query can scan on a single node. When you add more nodes, the data is distributed more thinly. A query that used to scan 100M rows on one node might now scan 50M rows on two nodes. * Result: These limits actually become "safer" (harder to hit) as you scale out, so you typically don't need to change them unless you are trying to tighten restrictions. Summary Table Parameter | Type | Scaling Action

concurrency_limit

| Absolute Integer | Increase to allow more simultaneous queries.

mem_limit

| Percentage | Keep same. (Total memory grows with node count).

cpu_weight

| Relative Ratio | Keep same.

max_cpu_cores

| Per-node Integer | Keep same (unless new hardware has more cores).

big_query_...

| Per-node Limit | Keep same (becomes less restrictive as data spreads). Note for v3.1.4+: StarRocks introduced collective concurrency management where the Leader FE manages the queue. Even with this centralized management, the

concurrency_limit

is the threshold the FE uses to decide when to queue. Therefore, you still must increase this number to allow the FE to let more queries through to your new CNs. References • administration/management/resource_management/query_queues.md • faq/operation_maintenance_faq.md • administration/management/resource_management/resource_group.md