In a Starrocks 4.0 shared data cluster, what metric can I look at to determine if a query is running? I have KEDA looking at prometheus metrics and I don't want to scale down if a query is running. I'm having trouble finding that metric.

starrocks_fe_connection_total

only shows me how many connections I have. If I connect via mysql that number jumps to 1, but running

SELECT sleep(300)

doesn't change it.

@Rocky - what sql can be used to see how many queries are running? We have resource groups setup, but need to just see if users in the group are running queries.

To see how many queries are currently running and identify which resource groups they belong to, you can use several different commands depending on your StarRocks version and how much detail you need. 1. The Most Comprehensive View (v3.4+) If you are on version 3.4 or later, use the global procedure view. This shows queries from all FE nodes in the cluster and explicitly includes the User and ResourceGroup names.

sql
SHOW PROC '/global_current_queries';

* Why use this: It's the easiest way to see the human-readable Resource Group name and the User side-by-side. 2. View Queries on the Current FE If you are on an older version or only care about the FE you are currently connected to:

sql
SHOW PROC '/current_queries';

* Columns to watch: Look at the

User

and

ResourceGroup

columns to verify if your specific group is active. 3. Using

SHOW RUNNING QUERIES

(v3.1.4+) This command is specifically designed to monitor the query queue and execution status across the cluster.

sql
SHOW RUNNING QUERIES;

* Columns to watch: * `ResourceGroupId`: Matches the ID of your resource group. * `State`: Shows

RUNNING

(currently executing) or

PENDING

(waiting in a queue). * Note: This displays the

ResourceGroupId

(numeric) rather than the name. 4. Basic Process List For a quick look at all active connections and their current SQL:

sql
SHOW PROCESSLIST;

* This is the standard MySQL-compatible way to see what's happening, though it doesn't explicitly label the Resource Group in the default view. Pro-Tip: Identifying Resource Group IDs If you use

SHOW RUNNING QUERIES

and need to know which ID belongs to which group name, you can query:

sql
SELECT * FROM information_schema.resource_groups;

Recommendation: Use

SHOW PROC '/global_current_queries'

for the most readable output containing both the username and the resource group name. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_RUNNING_QUERIES.md • administration/management/monitoring/metrics.md • using_starrocks/running_queries.md • administration/management/resource_management/query_queues.md

@Rocky - how can we see global running queries of a starrocks cluster in Prometheus?

To monitor global running queries of a StarRocks cluster in Prometheus, you should look for specific metrics exposed by the Frontend (FE) and Backend (BE) nodes. StarRocks exposes metrics in Prometheus format at the following endpoints: * FE:

http://<FE_IP>:<FE_HTTP_PORT>/metrics

(Default port:

8030

) * BE:

http://<BE_IP>:<BE_HTTP_PORT>/metrics

(Default port:

8040

) 1. Key Metrics for Running Queries The most relevant metrics for tracking currently active (running) queries are: *

starrocks_fe_unfinished_query

(FE Metric) * Description: Indicates the number of queries currently running (unfinished) on that specific FE node. * Type: Instantaneous. * Usage: Since each FE tracks the queries it is currently coordinating, you must aggregate this across all FEs to get a global view. *

pip_query_ctx_cnt

(BE Metric) * Description: The number of currently running query fragments/contexts on a BE node. * Usage: Useful for seeing the execution load across the storage/compute layer. *

starrocks_fe_query_queue_pending

(FE Metric) * Description: The number of queries currently waiting in the query queue (not yet running due to resource limits or concurrency control). 2. Prometheus Query for Global View To see the total global running queries across the entire cluster, use the following PromQL expression in your Prometheus or Grafana dashboard:

promql
sum(starrocks_fe_unfinished_query)

To see the total queued queries (pending) globally:

promql
sum(starrocks_fe_query_queue_pending)

3. Alternative SQL-based Global View If you want to see the actual details of these queries (not just the count) directly via SQL, StarRocks (starting from version 3.4) provides a global procedure:

sql
SHOW PROC '/global_current_queries';

For older versions, you can use

SHOW RUNNING QUERIES

to see the status of queries and whether they are in a

RUNNING

or

PENDING

state. Summary for Prometheus Configuration: Ensure your Prometheus

scrape_configs

includes all FE nodes. Once scraped, you can create a Grafana Singlestat or Time Series panel using

sum(starrocks_fe_unfinished_query)

to monitor your cluster's real-time query load. References • administration/management/monitoring/metrics.md • using_starrocks/running_queries.md • administration/management/resource_management/query_queues.md • administration/management/monitoring/Monitor_and_Alert.md

@Rocky I do not have a

starrocks_fe_unfinished_query

metric

The metric

starrocks_fe_unfinished_query

is primarily available in StarRocks v3.0 and later versions, particularly those that support the multi-warehouse (compute group) architecture. This metric indicates the number of queries currently running under a specific warehouse (or the

default_warehouse

in shared-nothing clusters). If you do not see this metric, it is likely due to one of the following reasons: 1. Version Compatibility If you are running StarRocks v2.5 or older, this specific metric name (

starrocks_fe_unfinished_query

) does not exist. In older versions, FE query metrics were primarily counters (like

starrocks_fe_query_total

). Recommended Alternative for older versions: To monitor current query load, you should check the BE-side metrics which provide a gauge of active queries: * Metric:

pip_query_ctx_cnt

(indicates the total number of currently running queries on a BE). * Metric:

starrocks_be_query_scan_bytes_per_second

(for load monitoring). 2. Multi-Warehouse Labels In v3.x, this metric often includes a

warehouse

label. If your Prometheus configuration filters out labels or expects a global metric without labels, it might not appear as expected. Check your FE metrics endpoint directly to see if it is present with labels:

bash
curl http://<FE_IP>:<FE_HTTP_PORT>/metrics | grep unfinished_query

3. Monitoring the Leader FE Some FE metrics are only reported by the Leader FE. If you are scraping a Follower FE, certain cluster-wide or coordination metrics might be missing or return

0

. Ensure you are scraping all FE nodes or checking the Leader. 4. Alternative Methods to View Running Queries If you need to see the current query count or details via SQL, you can use: *

SHOW PROC '/current_queries'

: Displays queries currently running on the FE node you are connected to. *

SHOW PROC '/global_current_queries'

(Available from v3.4+): Displays queries running across all FE nodes. *

SHOW RUNNING QUERIES

(Available from v3.1.4+): Specifically used to monitor query queue status and active executions. *

SHOW PROC '/backends'

: Check the

NumRunningQueries

column for each Backend to see the distributed load. If you are on v3.x and still don't see it, ensure that

enable_metric_calculator

is set to

true

in your

fe.conf

(it is

true

by default). References • administration/management/FE_configuration.md