http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> whats that query_queue_mem_used_pct...
# questions-and-troubleshooting
z
@Rocky whats that query_queue_mem_used_pct_limit,0.65 can you explain?
r
query_queue_mem_used_pct_limit
is a global system variable used to control StarRocks' Query Queue feature. Specifically, the value 0.65 means that when the memory usage on a Backend (BE) node exceeds 65% of its limit, StarRocks will stop executing new queries immediately and instead place them into a queue. How it works: * Threshold: It sets a memory "pressure" trigger. If a BE's memory usage is higher than 65%, the system assumes it is under heavy load. * Queuing: Instead of allowing a new query to start (which might cause an Out of Memory / OOM error), the query waits in a queue until memory usage drops below the threshold. * Safety: This prevents a surge of concurrent queries from crashing the BE nodes by exhausting all available RAM. Key Details: * Value Range: 
0
to 
1
(where 
0
means the limit is disabled). * Scope: This is a global variable. You can set it using:
Copy code
sql
    SET GLOBAL query_queue_mem_used_pct_limit = 0.65;
* Dependency: For this to take effect, the query queue feature must be enabled (usually via 
SET GLOBAL enable_query_queue = true;
). * Related Limits: It is often used alongside 
query_queue_concurrency_limit
(max number of concurrent queries) and 
query_queue_cpu_used_permille_limit
(CPU usage threshold). If you are seeing queries being queued or delayed with the message related to this limit, it means your BE nodes are currently using more than 65% of their configured memory. References • sql-reference/System_variable.md • administration/management/resource_management/query_queues.md • administration/management/BE_configuration.md • faq/resource_isolation_faq.md
z
@Rocky by default its enabled? in 3.5.16
r
In StarRocks (including version 3.5.16), the variable 
query_queue_mem_used_pct_limit
is not enabled by default. According to official documentation, the default value is 0, which means the memory-based queuing limit is disabled. For this limit to actually do anything, two things must happen: 1. The query queue feature itself must be enabled (e.g., 
SET GLOBAL enable_query_queue_select = true;
). 2. The variable must be set to a value greater than 0 (like the 0.65 you mentioned). Why do you see 0.65? While 0 is the factory default, 0.65 is a common manual setting used to prevent cluster instability. If your system has this set: * Threshold: It means that if a Backend (BE) node uses more than 65% of its allocated memory, any new incoming queries will be placed in a queue rather than running immediately. * Purpose: This prevents the cluster from crashing due to Out-of-Memory (OOM) errors during heavy load or concurrent "big" queries. Important Note for v3.3+: Starting from StarRocks v3.3 (which includes your 3.5.16 version), there is a newer feature called Query Queue v2 (
enable_query_queue_v2
). * If Query Queue v2 is enabled, the variable 
query_queue_mem_used_pct_limit
is ignored. * V2 uses a more advanced "slot-based" system to estimate query resources instead of relying on simple percentage thresholds. To check your current settings, you can run:
Copy code
sql
SHOW VARIABLES LIKE '%query_queue%';
References • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md • administration/management/resource_management/Big_query_analysis.md -analysis)
z
@Rocky I dont see any enable_query_queue_v2
r
You don't see 
enable_query_queue_v2
because it is an FE Configuration parameter, not a system variable. It won't appear in 
SHOW VARIABLES
. To check its status in StarRocks 3.5.16, you need to use the 
ADMIN SHOW CONFIG
command:
Copy code
sql
ADMIN SHOW CONFIG LIKE '%query_queue_v2%';
Why this matters for your version (3.5.16): * Legacy Mode (Default in 3.5): If 
enable_query_queue_v2
is 
false
(which is the default for version 3.5), the system uses the standard Query Queue. In this mode, 
query_queue_mem_used_pct_limit
is the primary way to trigger queuing based on memory usage. * V2 Mode: If you manually enable 
enable_query_queue_v2
, the system switches to a "slot-based" scheduler. This is a more modern approach that estimates the resource cost of a query (slots) before running it, rather than just reacting to the current memory percentage. To summarize the 0.65 setting: If you are using the default settings in 3.5.16 (where V2 is off), setting 
query_queue_mem_used_pct_limit
to 0.65 is the correct and active way to prevent memory-related crashes. It tells StarRocks: "If any BE node is using more than 65% memory, stop starting new queries and put them in the queue." How to enable the memory limit: 1. Check if query queuing is on for your query type:
Copy code
sql
    SHOW VARIABLES LIKE 'enable_query_queue_select';
2. If it's 
false
, enable it and set your limit:
Copy code
sql
    SET GLOBAL enable_query_queue_select = true;
    SET GLOBAL query_queue_mem_used_pct_limit = 0.65;
References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md References • sql-reference/System_variable.md • administration/management/resource_management/query_queues.md
z
@Rocky how can I set query_queue_v2_concurrency_level based on what?
r
In StarRocks v3.5, 
query_queue_v2_concurrency_level
is an FE Configuration parameter that determines the total number of "slots" available in your cluster for Query Queue V2. 1. The Formula The value you set is a multiplier for your cluster's total CPU cores. StarRocks calculates the total available query slots as follows: 
Total Slots = query_queue_v2_concurrency_level _ Number of BEs _ Cores per BE
 * Default: 4 * Example: If you have 3 BEs, each with 16 cores, and the level is set to 4: _ 
4 _ 3 * 16 = 192 total slots.
2. What to base the value on? You should set this based on the nature of your workload and your resource safety margins: * Memory Availability per Query: Each slot is allocated a portion of memory. _ 
Memory per Slot = BE Memory / (Cores per BE _ query_queue_v2_concurrency_level)
* Higher Level (e.g., 8 or 16): Allows more concurrent queries but gives each query less memory. Use this if you have many small, fast queries. * Lower Level (e.g., 2 or 4): Allows fewer concurrent queries but gives each query more memory. Use this if you run heavy, memory-intensive analytical queries. * Concurrency Needs: If you find that many queries are stuck in the queue even though CPU/Memory usage is low, you should increase this value to create more slots. * Resource Fragmentation: Do not set this value excessively high (e.g., 64+). If each slot has too little memory, queries may fail frequently or cause overhead due to extreme fragmentation. 3. How to set it Because it is an FE configuration, you change it using the 
ADMIN
command:
Copy code
sql
-- Check current value
ADMIN SHOW CONFIG LIKE '%query_queue_v2_concurrency_level%';

-- Set a new value (e.g., 8)
ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = "8");
Summary Recommendation: * Stick with the default (4) for most balanced workloads. * Decrease (1-2) if you are hitting OOM errors and want to force queries to wait for more memory. * Increase (8-12) if you have a high-concurrency dashboard with very small queries and a lot of queuing delay. References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md SET_CONFIG) References • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md • sql-reference/System_variable.md
z
@Rocky we have heavey queries so it better to keep 4?eventhough we have 28 cpu and 102 ram on each cn and we have 3 cns?
r
With 28 CPUs and 102 GB RAM per node (3 CNs), the default 
query_queue_v2_concurrency_level
of 4 creates a very granular slot system. For "heavy" queries, you might actually want to lower it to 1 or 2. Here is the breakdown based on your hardware: 1. Calculation for your Cluster (Level = 4) * Total Slots: $4 \times 3 \text{ CNs} \times 28 \text{ Cores} = \mathbf{336 \text{ slots}}$ * Memory per Slot: $102 \text{ GB} \div (28 \text{ Cores} \times 4) \approx \mathbf{0.91 \text{ GB/slot}}$ With these settings, StarRocks thinks a "standard" unit of work only needs 910 MB. If your queries are "heavy" (e.g., they often need 10GB–20GB each), the system will have to assign 11 to 22 slots per query. 2. Why lower it to 1 or 2 for heavy queries? If your workload consists of fewer, larger queries rather than hundreds of tiny ones, a level of 1 or 2 is safer: * Level 1 Calculation: Each slot becomes ~3.6 GB ($102 \div 28$). * Benefit: It forces the scheduler to be more conservative. By making each "slot" larger, you reduce the risk of the system over-committing memory. If many heavy queries hit at once, a lower concurrency level will trigger the queue sooner, preventing the Compute Nodes (CNs) from hitting OOM (Out of Memory) and crashing. 3. Recommendation * Keep at 4 if you have a "mixed" workload (many small dashboard queries + some heavy ones). * Change to 1 or 2 if your cluster is dedicated to heavy ETL or complex data science queries where memory is the primary bottleneck. Important: Is V2 actually on? In version 3.5.16, 
enable_query_queue_v2
is typically false by default (it becomes the default in later versions). * Check its status: 
ADMIN SHOW CONFIG LIKE '%enable_query_queue_v2%';
* If it is FALSE: The 
concurrency_level
setting is doing nothing. You should instead focus on 
query_queue_mem_used_pct_limit = 0.65
and 
query_queue_concurrency_limit
(the legacy system). * If you want to use the V2 features: You must first set 
ADMIN SET FRONTEND CONFIG ("enable_query_queue_v2" = "true");
. Summary: For heavy queries on your specific hardware, setting 
query_queue_v2_concurrency_level
to 1 or 2 provides a better "safety buffer" than the default of 4. References • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md • administration/management/resource_management/Big_query_analysis.md -analysis) References • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/management/resource_management/query_queues.md
z
@Rocky after setting that 
ADMIN SET FRONTEND CONFIG ("enable_query_queue_v2" = "true");
. do I need restart?
r
Yes, you must restart your FE nodes for that specific change to take effect. While many StarRocks parameters can be changed dynamically, 
enable_query_queue_v2
is classified as non-mutable at runtime. This is because enabling it triggers a fundamental switch in the FE's internal scheduling logic (moving from a reactive resource monitor to a proactive "slot-based" strategy), which is initialized during the FE startup process. Correct steps to enable it: 1. Modify the configuration file: Add the following line to the 
fe.conf
file on all FE nodes:
Copy code
properties
    enable_query_queue_v2 = true
2. Restart the FEs: Restart your FE services one by one (following your standard rolling upgrade/restart procedure). 3. Verify: After the restart, run this command to confirm it is active:
Copy code
sql
    ADMIN SHOW CONFIG LIKE '%enable_query_queue_v2%';
Why the 
ADMIN
command alone isn't enough: Even if the 
ADMIN SET FRONTEND CONFIG
command appears to succeed (depending on your specific sub-version), the underlying "Slot Manager" that handles Query Queue V2 will not start up or take over the scheduling until the process is fresh. Once enabled: Remember that after enabling V2, the legacy variables like 
query_queue_mem_used_pct_limit
will no longer be used. Instead, the system will use: * 
query_queue_v2_concurrency_level
(The one we discussed earlier). * 
SET GLOBAL enable_query_queue_select = true;
(This variable still needs to be true to actually put queries in the new V2 queue). References • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • using_starrocks/caching/query_cache.md
z
@Rocky running admin set config from starrocks wont take effect?
r
For the specific parameter 
enable_query_queue_v2
, running 
ADMIN SET FRONTEND CONFIG
will not take effect immediately because it is a non-mutable (static) configuration. Here is the breakdown of how these two settings behave: 1. 
enable_query_queue_v2
(Static) * Is mutable: No. * Behavior: This parameter is only read when the FE process starts. If you try to change it via 
ADMIN SET FRONTEND CONFIG
, the command might return "OK," but the internal "Slot Manager" (the engine for V2) will not actually initialize or start managing queries. * Requirement: You must add it to 
fe.conf
and restart the FE. 2. 
query_queue_v2_concurrency_level
(Dynamic) * Is mutable: Yes. * Behavior: Once V2 is already enabled (after your restart), you can use 
ADMIN SET FRONTEND CONFIG
to change this level (e.g., from 4 to 2) at runtime without a restart. The system will adjust the available slots and memory-per-slot calculations immediately. Summary of Actions To correctly switch to and tune Query Queue V2: 1. Update 
fe.conf
 on all FEs: 
enable_query_queue_v2 = true
. 2. Restart all FEs. 3. Verify it is on: 
ADMIN SHOW CONFIG LIKE 'enable_query_queue_v2';
. 4. Tune dynamically (no restart needed for these):
Copy code
sql
    -- If queries are too heavy, reduce the level to 1 or 2
    ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = "2");

    -- Ensure the global session variables are also enabled
    SET GLOBAL enable_query_queue_select = true;
References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md References • administration/management/FE_configuration.md • sql-reference/System_variable.md
4 Views
Open in Slack
PreviousNext
Powered by