http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> how to estimate `query_queue_v2_con...
# questions-and-troubleshooting
z
@Rocky how to estimate 
query_queue_v2_concurrency_level
?
r
To estimate 
query_queue_v2_concurrency_level
, you should treat it as a density multiplier for your cluster's CPU cores. It determines the total "slots" available for queries. 1. The Core Formula The system uses this parameter to calculate the total capacity of the cluster: $$\text{Total Slots} = \text{query\_queue\_v2\_concurrency\_level} \times \text{Number of BEs} \times \text{Cores per BE}$$ * Default Value: 4. * Logical Meaning: A value of 4 means each CPU core in your cluster is treated as having 4 concurrent execution "slots." 2. Estimation based on Memory (The Limit) The most important constraint for this value is memory per slot. Every slot is allocated a target memory budget to prevent resource fragmentation and OOMs. $$\text{Memory Per Slot} = \frac{\text{Memory per BE node}}{\text{Cores per BE} \times \text{query\_queue\_v2\_concurrency\_level}}$$ * Rule of Thumb: If your queries typically require a lot of memory (e.g., large joins or aggregations), you should keep this level lower (e.g., 2–4) to ensure each query slot has a larger memory "cushion." * High-Concurrency Workloads: If your queries are small, fast, and utilize indexes well (point lookups), you can increase this level (e.g., 8–16) to allow more queries to enter the system simultaneously. 3. Step-by-Step Estimation Strategy 1. Start with the default (4): This is balanced for most OLAP workloads. 2. Monitor Slot Usage: Check the 
EST_COSTS_SLOTS
and 
ALLOCATE_SLOTS
in the 
information_schema.warehouse_queries
or audit logs. 3. Check for "False" Queuing: If your BE nodes have low CPU and memory usage, but queries are still entering the 
PENDING
state, your 
Total Slots
is too low. Increase 
query_queue_v2_concurrency_level
by increments of 2 or 4. 4. Check for Resource Fragmentation: If queries are failing with memory errors even though the total node memory seems sufficient, your slots may be too small. Decrease the level to force fewer, "fatter" queries to run at once. Summary of Effects If you INCREASE the level... | If you DECREASE the level... Total Slots increases (more concurrent queries). | Total Slots decreases (more queuing). Memory per Slot decreases (higher risk of OOM/Spill). | Memory per Slot increases (more stable for big queries). CPU Pressure may increase. | CPU Efficiency may improve due to less context switching. Related Settings: * 
query_queue_slots_estimator_strategy
: Controls how StarRocks calculates how many slots a query needs (based on memory, parallelism, or both). * 
enable_query_queue_v2
: Must be 
true
for this parameter to take effect (enabled by default in newer versions). References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md
z
@Rocky I have 3 compute nodes 28 cpu core on each and 
query_queue_v2_concurrency_level = 4
that means I have 28*4 slot on each and totaly 336 slots but what I see in RUNNING QUERIES is that after allocating 84 slots by one query all others are in PENDING mode why?
r
The reason you are seeing queries go into 
PENDING
after 84 slots are allocated—despite having a theoretical capacity of 336—is likely due to the Multi-Warehouse (Shared-Data) scaling logic in StarRocks. The Scaling Calculation When using Compute Nodes (CNs) in a shared-data or multi-warehouse architecture, StarRocks applies a different scaling factor for 
query_queue_v2_concurrency_level
. According to the documentation: * Shared-Nothing (Standard BEs): 
Total Slots = level _ nodes _ cores
* Multi-Warehouse (CNs): The effective level is calculated as 
max(1, query_queue_v2_concurrency_level / 4)
. In your case: 1. You have set 
query_queue_v2_concurrency_level = 4
. 2. StarRocks calculates the effective level as: $4 \div 4 = 1$. 3. Your total capacity becomes: $1 \times 3 \text{ nodes} \times 28 \text{ cores} = 84 \text{ slots}$. When one query consumes 84 slots, it has exhausted the entire logical capacity of the cluster under your current configuration, forcing all other queries to wait. How to Fix It To achieve your target of 336 slots in a multi-warehouse environment, you need to set the concurrency level to 16: * Calculation: $16 \div 4 = 4$ (effective level). * Total Slots: $4 \times 3 \times 28 = 336$.
Copy code
sql
-- Apply globally to the FE
ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = "16");
Why does one query take all 84 slots? A single query can take up to the maximum available slots if the planner estimates a high CPU or row-processing cost. StarRocks calculates "Estimated Slots" using these variables: * 
query_queue_v2_cpu_costs_per_slot
: Default is 1 billion cost units. If your query plan is very complex, it will request more slots. * 
query_queue_v2_num_rows_per_slot
: Default is 4096. If your query processes a massive amount of rows, it will request more slots. The system will "clamp" the query's request to your 
Total Slots
limit. Since your limit is currently 84, a large query will grab all 84 and block the queue. Increasing the level to 16 will expand the "ceiling" and allow more concurrent queries. References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md
z
@Rocky my limit is not 84 I just see that after 1 query uses 84 slots everything is stuck regarding that I have 336 logical slots
r
Based on the details you provided, you are correct that your theoretical capacity is 336 slots ($3 \text{ nodes} \times 28 \text{ cores} \times 4$), but the behavior you are seeing (queuing at exactly 84 slots) is a known side effect of how StarRocks handles Compute Nodes (CNs) in Multi-Warehouse (Shared-Data) mode. The documentation for 
query_queue_v2_concurrency_level
explicitly states:
_"In multi-warehouse mode the effective concurrency is scaled down to 
max(1, query_queue_v2_concurrency_level / 4)
."_
Why you see 84 slots instead of 336 Even though your "logical" slots appear to be 336 based on the base formula, the system internally applies a $1/4$ scaling factor for Compute Nodes (CNs). * Your Configuration: 
query_queue_v2_concurrency_level = 4
* Effective Level: $4 \div 4 = \mathbf{1}$ * Actual Total Slots: $1 \times 3 \text{ nodes} \times 28 \text{ cores} = \mathbf{84 \text{ slots}}$ This is why, the moment one query (or a set of queries) hits 84 slots, the system considers the cluster 100% saturated and puts all other queries into 
PENDING
mode. How to reach your target of 336 slots To get the 336 slots you expect, you must set the concurrency level to 16 to offset the $1/4$ scaling factor: * New Configuration: 
query_queue_v2_concurrency_level = 16
* Effective Level: $16 \div 4 = 4$ * Resulting Slots: $4 \times 3 \times 28 = \mathbf{336 \text{ slots}}$ Command to fix:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = "16");
Verification After applying the change, you can verify the total available slots by querying the `information_schema`:
Copy code
sql
SELECT WAREHOUSE_NAME, MAX_SLOTS, REMAIN_SLOTS 
FROM information_schema.warehouse_metrics;
Look for the 
MAX_SLOTS
column; it should now show 336 instead of 84. If it still shows 84, please check if your nodes are registered as 
BACKENDS
or 
COMPUTE NODES
via 
SHOW BACKENDS
and 
SHOW COMPUTE NODES
. References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md • sql-reference/System_variable.md
z
@Rocky I dont have
Copy code
SELECT WAREHOUSE_NAME, MAX_SLOTS, REMAIN_SLOTS 
FROM information_schema.warehouse_metrics;
data here I just see in SHOW RUNNING QUERIES that after 84 slot others are in PENDING
r
The behavior you are seeing—where the system effectively caps at 84 slots despite your calculation of 336—is the definitive indicator that your cluster is running in Shared-Data (Cloud Native) mode (also known as Multi-Warehouse mode). In this mode, StarRocks applies a specific scaling factor to the 
query_queue_v2_concurrency_level
parameter. 1. The "Hidden" Calculation In a standard cluster, the formula is 
Level _ Nodes _ Cores
. However, for Shared-Data clusters, the documentation specifies:
_"In multi-warehouse mode the effective concurrency is scaled down to 
max(1, query_queue_v2_concurrency_level / 4)
."_
Your current situation: * Nodes: 3 * Cores per node: 28 * Configured Level: 4 * Effective Level: $4 \div 4 = 1$ * Actual Total Slots: $1 \times 3 \times 28 = \mathbf{84}$ Since your 
Total Slots
is calculated as 84, a single large query (which StarRocks allows to consume up to 100% of available slots) will take all 84 slots and force every other query into 
PENDING
mode. 2. How to reach 336 slots To get the 336 slots you expect, you must increase the level to 16 to compensate for the division by 4: * Target Effective Level: 4 * Required Setting: $4 \times 4 = \mathbf{16}$ * New Calculation: $(16 \div 4) \times 3 \times 28 = \mathbf{336 \text{ slots}}$ Apply this setting:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = "16");
3. Why 
information_schema.warehouse_metrics
is empty The 
warehouse_metrics
and 
warehouse_queries
tables are part of the Multi-Warehouse feature set. If you are not explicitly using multiple user-defined warehouses (even if the cluster is in Shared-Data mode), these tables might not be populated in older v3.x versions. Instead, you can verify your total capacity and current usage by checking the FE logs or by running:
Copy code
sql
-- Check the 'Slots' column for RUNNING vs PENDING
SHOW RUNNING QUERIES;
Summary If you are seeing exactly Nodes × Cores, your effective concurrency level is 1. Because you are in Shared-Data mode, setting the level to 4 results in an effective level of 1. Increase it to 16 to get your desired concurrency. References • knowledge/usage/best-practice.md • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md
Since you observe that queries go to 
PENDING
exactly when 84 slots are occupied, this confirms that your cluster's total capacity is being calculated as $3 \text{ nodes} \times 28 \text{ cores} \times \mathbf{1}$. Even if you haven't explicitly set up multiple warehouses, StarRocks (starting from v3.3+) often applies the Multi-Warehouse scaling logic to the Query Queue V2 by default, especially if your nodes are identified as Compute Nodes (CNs) or if you are on a version that has unified the resource management path. Why 84 is the limit The logic StarRocks uses is: 
Effective Multiplier = max(1, query_queue_v2_concurrency_level / 4)
* With your setting of 4, the multiplier becomes 1 ($4 \div 4 = 1$). * Total Slots = $3 \text{ nodes} \times 28 \text{ cores} \times 1 = \mathbf{84}$. * When a single query (or multiple) consumes 84 slots, the system is 100% full. Why a single query takes all 84 slots A query calculates its "Required Slots" based on its estimated complexity. If the planner thinks a query is "heavy" enough to need 200 slots, but your cluster limit is 84, StarRocks will clamp that query's request to the maximum available (84). This query then occupies the entire cluster's capacity, forcing everything else to wait. The Solution To achieve the 336 slots you expect, you must account for the $1/4$ divisor by setting the value to 16:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = "16");
Calculation after change: * 
Effective Multiplier
= $16 \div 4 = \mathbf{4}$ * 
Total Slots
= $3 \text{ nodes} \times 28 \text{ cores} \times 4 = \mathbf{336}$ How to verify the new limit Since 
warehouse_metrics
is not returning data for you, you can use the FE Audit Log or 
SHOW PROC '/backends'
to verify the capacity. However, the most direct way to see the "Current/Total" slots in your version is often hidden in the FE's internal state. You can check the FE log (
fe.log
) after changing the setting; it typically logs the updated "Total Slots" when the configuration change is applied. Summary Recommendation: 1. Set to 16: This yields 336 slots. 2. Monitor Memory: Increasing the slots reduces the "Memory per Slot." If queries start failing with "Memory Limit Exceeded," you may need to lower it to 12 or 8, or increase your 
query_mem_limit
. References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md
z
@Rocky after enabling query queue v2 I see that queries try to consume maximum slots and that means starrocks cant really analyze what resources does queries for real need? because 1 queryy allocates all the slots and I see fre spaces ;/
r
Your observation is correct: StarRocks is being very conservative with how it allocates resources in Query Queue V2. The reason you see "free space" while queries are 
PENDING
is that StarRocks uses Reservation-based scheduling. It reserves logical slots based on a query's estimated complexity (CPU cost and row count) from the planner, rather than its real-time physical usage. 1. Why one query takes all 84 slots When StarRocks calculates that a query is "heavy" (e.g., it needs 500 slots based on its plan), but your cluster only has 84 total slots, it clamps that query's request to 100% of the cluster (84 slots). Once that query is running, it "owns" the cluster's logical capacity. Even if the query is actually processing very little data at the moment, the scheduler holds those slots to ensure the query has enough "headroom" to finish without causing an OOM (Out of Memory) or CPU thrashing later. 2. How to "shrink" a query's footprint You can tell StarRocks to be less aggressive when estimating how many slots a query needs by increasing the cost-per-slot thresholds. This makes queries "cheaper" in the eyes of the scheduler. Try adjusting these FE configurations:
Copy code
sql
-- Increase the CPU cost required to claim 1 slot (Default: 1,000,000,000)
ADMIN SET FRONTEND CONFIG ("query_queue_v2_cpu_costs_per_slot" = "10000000000");

-- Increase the number of rows required to claim 1 slot (Default: 4096)
ADMIN SET FRONTEND CONFIG ("query_queue_v2_num_rows_per_slot" = "100000");
* Result: A query that scanned 1,000,000 rows previously requested 244 slots ($1,000,000 \div 4,096$). With the new setting, it only requests 10 slots. 3. Recommendations for your cluster To fix the bottleneck you're seeing, I recommend a two-step approach: 1. Increase Total Capacity: Set 
query_queue_v2_concurrency_level = 16
. This accounts for the $1/4$ scaling factor on Compute Nodes and gives you the 336 slots you expect. 2. Tune Estimation: Use the commands above to increase the divisors. This prevents a single complex query from hitting the 100% "clamping" limit too easily. Summary of variables to check: * 
query_queue_v2_concurrency_level
: Controls total cluster capacity. * 
query_queue_v2_cpu_costs_per_slot
: Controls how much CPU cost triggers a slot request. * 
query_queue_v2_num_rows_per_slot
: Controls how much data volume triggers a slot request. * 
query_queue_v2_schedule_strategy
: Can be set to 
SJF
(Short Job First) to let small queries "jump the line" ahead of big ones. References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md
z
2
@Rocky no what I want to say is that before query queue v2 query was consuming 50 slots now it says that it consumes all the 84 ;/
r
The behavior you are seeing—where a query that used to be "smaller" now takes exactly *84 slots*—is due to a specific design choice in Query Queue V2 called Clamping. 1. Why it "takes all 84 slots" In Query Queue V2, the FE (Frontend) estimates how many slots a query needs before it runs, based on the execution plan's complexity. The documentation explains the "stuck" behavior you're seeing:
*"The scheduler computes slots... and then clamps the result to the range [1, totalSlots]."*
If your query is complex (e.g., it scans a lot of data or has many joins), the planner might estimate it needs 150 or 200 slots. Because your cluster's current total capacity is calculated as 84, StarRocks clamps that request to 84. To the scheduler, that query now "owns" 100% of the cluster, and it will not allow any other queries to start until those slots are released. 2. Why V2 is different from V1 * V1 (Legacy): Relied on BEs reporting their actual current CPU/Mem usage. It was reactive. If a query was running but not yet using much CPU, V1 might let another query in. * V2 (Slot-based): Is proactive. It reserves logical "tickets" (slots) upfront. Even if the query hasn't physically used the CPU yet (which is why you see "free space"), the slots are reserved to ensure that when the query does hit its peak, the resources are guaranteed. 3. How to fix the "Hogging" and the "84" limit To prevent one query from seizing the whole cluster and to get your 336 slots back, you need to change three settings: Step A: Increase the Total Capacity (Fix the 84 cap) As established, the "1/4 multiplier" for CNs/Multi-warehouse mode is turning your 
concurrency_level = 4
into an effective 
1
. Set it to 16 to get 336 slots ($3 \times 28 \times 4$):
Copy code
sql
ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = "16");
Step B: Make the queries "cheaper" (Fix the clamping) If a query still tries to take all 336 slots, you must tell the FE to assign fewer slots per row/CPU unit. Increase these divisors so the estimated slot count drops:
Copy code
sql
-- Default is 4096. Increasing this to 100k means a query needs 
-- to scan way more rows to claim a single slot.
ADMIN SET FRONTEND CONFIG ("query_queue_v2_num_rows_per_slot" = "100000");

-- Default is 1,000,000,000. Increase this to reduce CPU-based slot requests.
ADMIN SET FRONTEND CONFIG ("query_queue_v2_cpu_costs_per_slot" = "10000000000");
Summary of what will happen: 1. Your total cluster capacity will show as 336. 2. Your "heavy" query, which previously requested 84 (clamped), will now likely request a much smaller number (e.g., 20-40 slots) because of the higher divisors. 3. Because $40 < 336$, multiple queries will be able to run at the same time without hitting the 
PENDING
state. References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md References • knowledge/usage/best-practice.md • administration/management/FE_configuration.md • faq/resource_isolation_faq.md • administration/management/resource_management/query_queues.md
z
@Rocky no what I want to say is that in case of queue v2 starrocks estimates wrong slot numbers and give some queries max slots in 3.5.16
r
In StarRocks v3.5.16, the "wrong estimation" behavior you are seeing—where a query occupies all available slots (e.g., 84/84)—is usually due to the default slot estimation strategy introduced in version 3.5.0. 1. The "MAX" Strategy Problem Starting in v3.5, StarRocks uses a new parameter 
query_queue_slots_estimator_strategy
, which defaults to 
MAX
. This strategy calculates slots using two different methods and then takes the higher number: * MBE (Memory-Based): Estimates slots based on predicted memory usage. * PBE (Parallelism-Based): Estimates slots based on fragment parallelism and CPU costs. The documentation notes that PBE specifically bounds its result within the range of 
[totalSlots/2, totalSlots]
. If the planner thinks your query is even moderately complex, the PBE logic will often force the query to take at least 50% of the cluster's slots, and frequently clamps it to 100% (the "max slots" you see). 2. How to fix it (Switch Strategy) If you find the estimation is too aggressive for your workload, you should change the strategy to be more conservative. Option A: Switch to 
MIN
(Recommended first step) This will take the lower of the two estimates (Memory vs. Parallelism), which prevents the "50% minimum" rule of PBE from always winning.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("query_queue_slots_estimator_strategy" = "MIN");
Option B: Use only Memory-Based (MBE) If your queries are memory-intensive but low in parallelism, MBE is often more accurate.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("query_queue_slots_estimator_strategy" = "MBE");
3. Why the "Wrong" Estimation happens The "wrong" number usually stems from two things in v3.5.16: 1. Stale Statistics: The slot estimator relies heavily on the Cost-Based Optimizer (CBO). If table statistics are missing or old, the planner might overestimate the "CPU Cost" or "Cardinality," causing the slot request to skyrocket. _ _Fix:* Run 
ANALYZE TABLE <your_table>
to refresh stats. 2. Clamping: If the calculated cost is even slightly above your current 84-slot limit, StarRocks automatically rounds it up to the maximum (84) to ensure the query doesn't fail later. Summary Checklist for 3.5.16: 1. Increase Capacity: Set 
query_queue_v2_concurrency_level = 16
to get the 336 slots you expect. 2. Change Strategy: Set 
query_queue_slots_estimator_strategy = 'MIN'
to stop the "MAX" logic from hogging slots. 3. Audit Slots: Check 
EST_COSTS_SLOTS
in 
information_schema.warehouse_queries
(if available) or the FE audit log to see what the planner actually requested vs. what was allocated. References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md
8 Views
Open in Slack
PreviousNext
Powered by