@Rocky I have a staging service querying production starrocks for e2e testing. I need to contain the cpu it uses. How should I go about it? Right now, I'm thinking of giving it only 4/32 cores per CN node. It also has concurrency limit of 25. Does this mean all 25 concurrent queries (at limit) will be sharing 4 reserved cpu? Should I change concurrency limit to 1 and do query queueing and increase the staging e2e timeout instead?

To contain the CPU usage of your staging service on a production StarRocks cluster, you should use Resource Groups. Here is how the mechanics work and the recommended approach for your scenario: 1. Will 25 queries share the 4 cores? Yes. If you set a resource group with a limit of 4 cores (using

cpu_core_limit

or

exclusive_cpu_cores

) and a

concurrency_limit

of 25, StarRocks will allow 25 queries to run simultaneously within that group. They will all compete for the same 4 CPU cores on each CN/BE node. This usually results in high latency for all 25 queries because the CPU becomes a bottleneck, and the system overhead of context switching increases. 2. Hard vs. Soft Limits * Soft Limit (

cpu_core_limit

/

cpu_weight

): If the production cluster is idle, the staging service can exceed the 4-core limit. It only gets restricted to 4 cores when the production workload is also active and competing for CPU. * Hard Limit (

exclusive_cpu_cores

): Available in newer versions (v3.1.4+). This reserves 4 cores exclusively. Staging will never use more than 4 cores, even if the rest of the cluster is idle. This is the best way to ensure zero impact on production. 3. Recommended Strategy: Queuing and Concurrency Changing the

concurrency_limit

to a lower number (like 1 or 2) and enabling Query Queuing is a much better practice for staging/E2E tests: * Better Performance: If

concurrency_limit

is 1, that single query can utilize the 4 allocated cores more efficiently (set

pipeline_dop

to 4 for that session). * Stability: Instead of 25 queries running slowly and potentially timing out, they will wait in a queue and run one by one at full speed. * How to enable queuing: 1. Ensure you are on a version that supports group-level queuing (v3.1.4+). 2. Set

concurrency_limit = 1

in your resource group. 3. Enable the queue by setting the session variable:

SET GLOBAL enable_group_level_query_queue = true;

. 4. Adjust

query_queue_max_queued_queries

(default is 1024) and

query_queue_pending_timeout_second

(default is 300s) to allow your 25 queries to wait without failing. Summary Checklist 1. Use

exclusive_cpu_cores = 4

if you want a strict hard cap to protect production. 2. Lower

concurrency_limit

(e.g., to 1 or 2) to prevent internal thrashing. 3. Enable

enable_group_level_query_queue

so that the extra queries wait instead of failing with a "concurrency limit exceeded" error. 4. Increase E2E timeouts in your staging service to account for the time spent waiting in the StarRocks queue. For more details on syntax, see the Resource Group and Query Queue documentation. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md

@Rocky So query queueing helps by reducing cpu contention so queries can complete more quickly (total time needed) instead of constantly competing for resources? How does pipeline dop help?

Yes, you've got the right intuition. Here is a breakdown of how queuing and

pipeline_dop

work together to solve your problem: 1. How Query Queuing Helps When you have 25 queries sharing 4 cores, each query is essentially getting a "tiny slice" of CPU time before being swapped out (context switching). * Without Queuing: All 25 queries start immediately. They compete for the 4 cores, leading to massive CPU scheduling overhead. Total execution time for the batch increases because the CPU spends too much time managing the "traffic" rather than doing actual work. * With Queuing: By setting

concurrency_limit = 1

or

2

, you ensure that only 1 or 2 queries are active on those 4 cores at any given time. This eliminates contention. The "Total time to finish 25 queries" is almost always lower when queued because each query runs to completion without being interrupted by 24 others. 2. How

pipeline_dop

Helps

pipeline_dop

(Degree of Parallelism) controls how many threads a single query uses on each BE/CN node. * The Default Behavior: If

pipeline_dop

is set to

0

(default), StarRocks automatically sets it—usually to half the total cores of the machine. On a 32-core node, a single query might try to spawn 16 threads. * The Conflict: If your Resource Group only allows 4 cores, but the query tries to run 16 threads, those 16 threads will fight for the 4 allowed cores. This causes internal "thrashing" within the resource group. * The Solution: You should set

pipeline_dop

to match your resource limit. * If you set

exclusive_cpu_cores = 4

, set

pipeline_dop = 4

(via session variable or resource group property). * This ensures the query's internal parallelism matches the physical resources you've allocated, leading to the most efficient execution. Recommended Configuration To achieve the best results for your staging service: 1. Resource Group:

sql
    CREATE RESOURCE GROUP staging_group
    PROPERTIES (
        "exclusive_cpu_cores" = "4", -- Hard limit to protect Prod
        "concurrency_limit" = "1",    -- Run 1 at a time for max efficiency
        "mem_limit" = "10%"           -- Limit memory too
    );

2. Enable Queuing (Global or Session):

sql
    SET GLOBAL enable_group_level_query_queue = true;

3. Align Parallelism: When your staging service connects, have it run:

sql
    SET pipeline_dop = 4;

Summary: Queuing manages the external pressure (how many queries start), while

pipeline_dop

manages the internal pressure (how many threads each query uses). Aligning both to your 4-core limit will make your E2E tests much more stable without impacting production. References • faq/resource_isolation_faq.md • sql-reference/System_variable.md • knowledge/usage/best-practice.md • faq/Sql_faq.md

@Rocky What happens if query queieng is not enabled

If query queuing is not enabled (i.e.,

enable_group_level_query_queue = false

), any query that exceeds your

concurrency_limit

will fail immediately. Instead of waiting for a resource "slot" to open up, the user or service will receive an error like:

ERROR 1064 (HY000): Resource group staging_group concurrency limit exceeded

Summary of Behaviors: Feature Status | Behavior when Concurrency Limit is hit | Result Queue Disabled | New queries are rejected immediately. | Failures in your staging tests. Queue Enabled | New queries are put into a "Pending" state. | Success, but with increased latency. Why this matters for your E2E tests: If you set your

concurrency_limit

to 1 to prevent CPU contention but don't enable queuing, your staging service will likely crash or report errors as soon as it tries to run a second test in parallel. Recommendation: Always enable

enable_group_level_query_queue = true

if you are using low concurrency limits. This transforms a "Hard Failure" into "Wait Time," which is much easier to manage in a staging environment by simply increasing the timeout on your test runner. References • faq/resource_isolation_faq.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md References • faq/resource_isolation_faq.md • administration/management/resource_management/query_queues.md