@Rocky all the queries in SHOW RUNNING QUERIES are in pending status

When you see all queries in the

PENDING

status in

SHOW RUNNING QUERIES

, it means they are stuck in the Query Queue. This happens because the cluster has reached a configured resource or concurrency threshold and is preventing new queries from starting to avoid system overload. Here is how to troubleshoot and resolve this: 1. Check Query Queue Settings First, verify if query queues are enabled and what the limits are:

sql
SHOW VARIABLES LIKE 'enable_query_queue%';
SHOW VARIABLES LIKE 'query_queue%';

*

enable_query_queue_select

: If

true

, SELECT queries will queue when limits are hit. *

query_queue_concurrency_limit

: If this is set to a low number (e.g., 5), and you have 5 queries running, the 6th will be

PENDING

. *

query_queue_mem_used_pct_limit

: If BE memory usage exceeds this (e.g., 0.9 for 90%), all new queries will stay

PENDING

. *

query_queue_cpu_used_permille_limit

: If CPU usage exceeds this, queries will queue. 2. Check Backend Resource Usage Compare the limits from step 1 with the actual resource usage reported by the Backends (BEs):

sql
SHOW PROC '/backends'\G

Look for: *

MemUsedPct

: Is it higher than

query_queue_mem_used_pct_limit

? *

CpuUsedPct

: Is it higher than

query_queue_cpu_used_permille_limit

? *

NumRunningQueries

: Does it match or exceed your

concurrency_limit

? If the BEs are reporting high resource usage, StarRocks will keep all new queries in

PENDING

until usage drops. 3. Check Resource Groups If the queries in

SHOW RUNNING QUERIES

have a

ResourceGroupId

(other than

-

), they might be hitting limits specific to a Resource Group:

sql
-- View resource group configuration
SHOW RESOURCE GROUPS;

-- View real-time resource group usage
SHOW USAGE RESOURCE GROUPS;

Check if the

concurrency_limit

or

max_cpu_cores

for that specific group has been reached. 4. Common Causes & Solutions * Stuck "Running" Queries: Sometimes queries might be "leaked" (shown as RUNNING but not actually doing anything), occupying slots. You can try to kill them:

KILL QUERY <query_id>;

. * Limits too Aggressive: If your BEs are healthy but limits are set too low (e.g., memory limit at 0.5), queries will queue unnecessarily. You can increase them:

sql
    SET GLOBAL query_queue_concurrency_limit = 1000; -- Increase or set to 0 to disable
    SET GLOBAL query_queue_mem_used_pct_limit = 0.95;

* BE Reporting Issue: If a BE is down or disconnected, the FE might still think queries are "running" there. Check

Alive

status in

SHOW PROC '/backends';

. Summary Checklist 1. Run

SHOW PROC '/backends';

to see if CPU/Memory is genuinely high. 2. If resources look fine, check if

query_queue_concurrency_limit

is set too low. 3. If you want to bypass the queue immediately, set

SET GLOBAL enable_query_queue_select = false;

(use with caution as this may overload the system). References • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • administration/management/monitor_manage_big_queries.md • administration/management/FE_configuration.md

@Rocky difference between SHOW RUNNING QUERIES and SHOW PROC '/global_current_queries';

Both commands are used to monitor active queries, but they serve different purposes and provide different levels of detail regarding resource consumption and queuing. *1. SHOW RUNNING QUERIES* This command is primarily used to monitor the Query Queue status. * Focus: Queuing and scheduling. * Key Information: It shows if a query is

PENDING

(waiting in a queue) or

RUNNING

. * Unique Fields: * Slots: The logical resource capacity used by the query. * PendingTimeout: When a queued query will time out if it doesn't start. * ResourceGroupId: The specific resource group the query is hitting. * Use Case: Use this when you suspect queries are stuck or being throttled by concurrency limits. *2. SHOW PROC '/global_current_queries' (v3.4+)* This command provides a cluster-wide view of query resource consumption. * Focus: Real-time performance and resource usage. * Key Information: It aggregates information from all FE nodes in the cluster. * Unique Fields: * MemoryUsage: Current RAM used by the query. * CPUTime: Total CPU time consumed. * ScanBytes / ScanRows: How much data the query has actually processed so far. * DiskSpillSize: If the query is spilling to disk due to memory pressure. * Use Case: Use this to find "heavy" queries that are consuming too many resources (CPU/Memory) or scanning too much data. ────────── Comparison Summary Feature |

SHOW RUNNING QUERIES

|

SHOW PROC '/global_current_queries'

Primary Goal | Monitor the Query Queue | Analyze resource consumption Visibility | Global (all FEs) | Global (all FEs) Resource Metrics | No (only Slots) | Yes (CPU, Mem, Scan Rows/Bytes) Queue Info | Yes (Pending status/timeouts) | Limited (only

ExecState

) Supported Since | v3.1.4 | v3.4.0 (v2.5+ for

/current_queries

) Pro-tip: If you are on an older version (pre-3.4) and want to see queries across the whole cluster, you would have to run

SHOW PROC '/current_queries'

on each individual FE. References • using_starrocks/running_queries.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • using_starrocks/running_queries.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md • administration/management/resource_management/query_queues.md

@Rocky why queries can be showed in global current queries and not in show running queries? or I see running queries in global current queries and pending state on all show running queries?

The discrepancy you are seeing between

SHOW RUNNING QUERIES

and

SHOW PROC '/global_current_queries'

usually comes down to which system component is reporting the status and what types of queries each command tracks. 1. Why queries appear in

global_current_queries

but NOT in

SHOW RUNNING QUERIES

SHOW RUNNING QUERIES

is not a list of all active database connections; it is a specialized view for the Query Queue. * Query Queue Dependency:

SHOW RUNNING QUERIES

only displays queries that are subject to the StarRocks Query Queue logic. If

enable_query_queue_select

is set to

false

, or if the query is a "short" metadata query (like

SHOW

or

DESC

) that doesn't consume BE resources, it will never enter the queue and thus will not show up in

SHOW RUNNING QUERIES

. * Query Type:

global_current_queries

shows every query currently active on the Frontends (FEs), including metadata operations, internal tasks, and DDLs.

SHOW RUNNING QUERIES

focuses on data-processing queries (typically

SELECT

or

INSERT

) that compete for BE CPU/Memory resources. 2. Why status is "RUNNING" in one and "PENDING" in the other This happens because the two commands define "Running" differently: *

SHOW PROC '/global_current_queries'

(FE Perspective): In this view, a query is often marked as

RUNNING

(or

OK

) as soon as the FE has accepted the connection and started processing/parsing/planning the query. From the user connection perspective, the query is "in progress." *

SHOW RUNNING QUERIES

(Queue Manager Perspective): This command reports the dispatch state. A query is

PENDING

here if the FE has finished planning it but is holding it in the queue because the cluster-wide concurrency or memory limits have been reached. In short: The query is "Running" on the FE (planning/waiting), but "Pending" for the BEs (not yet allowed to execute). Summary Table Feature |

SHOW PROC '/global_current_queries'

|

SHOW RUNNING QUERIES

Scope | All FE activity | Only Queued/Resource-managed queries Status Meaning | Is the FE connection active? | Is the query dispatched to BEs? Common Values |

RUNNING

,

PENDING

(v3.4+) |

RUNNING

,

PENDING

Primary Use | Monitoring resource usage (Mem/CPU) | Monitoring concurrency bottlenecks Troubleshooting Steps If all queries are

PENDING

in `SHOW RUNNING QUERIES`: 1. Check if your resource limits are too tight:

SHOW VARIABLES LIKE 'query_queue%';

. 2. Check BE health and usage:

SHOW PROC '/backends';

. If

MemUsedPct

is higher than the queue limit, all queries will stay

PENDING

. 3. If you want to clear the queue immediately to test, you can set

SET GLOBAL enable_query_queue_select = false;

. References • using_starrocks/running_queries.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md • administration/management/resource_management/query_queues.md References • using_starrocks/running_queries.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md