@Rocky how can i monitor the queue depth of my compute nodes so i can understand how to support higher qps

To monitor the queue depth and identify bottlenecks for higher QPS on StarRocks compute nodes (CN/BE), you should look at two different levels of queuing: Query Queuing (at the entry point) and Internal Execution Queuing (at the engine level). 1. Monitor Query Queuing (Frontend Level) If you have enabled Query Queues (via

query_queue_concurrency_limit

or Resource Groups), queries will wait before execution if the system is at capacity. * Prometheus Metrics: * `starrocks_fe_query_queue_pending`: The current number of queries waiting in the global queue. * `starrocks_fe_resource_group_query_queue_pending`: The number of queries waiting for a specific resource group (useful for isolating workloads). * SQL Commands: * `SHOW RUNNING QUERIES`: Look for queries with

State = 'PENDING'

. * `SHOW PROCESSLIST`: Check the

IsPending

column; if

true

, the query is queued. * Audit Logs: * Check

fe.audit.log

for the

PendingTimeMs

field. High values indicate queries are spending too much time waiting for a slot. 2. Monitor Internal Execution Queuing (Compute Node Level) Even if queries are "running," they may be queued internally within the BE/CN execution engine. Monitoring these metrics is critical for understanding CPU and I/O saturation. * Pipeline Driver Queue: * Metric:

pipe_driver_queue_len

(or

starrocks_be_pipe_driver_queue_len

) * What it means: The number of "Ready" drivers waiting for an available CPU core. If this number is consistently high, your CPU is saturated, and increasing QPS will likely require more nodes or more cores per node. * Scan Executor Queuing: * Metric:

pipe_scan_executor_queuing

* What it means: The number of pending asynchronous I/O tasks. A high value suggests that your storage (disk/S3) cannot keep up with the scan requests. * Prepare Pool Queue: * Metric:

starrocks_be_pipe_prepare_pool_queue_len

* What it means: The number of tasks waiting to be initialized. High values can occur during bursts of many small, complex queries. 3. Key BE Process Metrics You can quickly check the real-time load on compute nodes using:

sql
SHOW PROC '/backends';

Look at: * NumRunningQueries: Number of active queries the BE is currently processing. * CpuUsedPct: If this is consistently near 100%, internal queues will naturally grow. Recommendations for Higher QPS * If

pipe_driver_queue_len

is high: You are CPU bound. Consider adding more CN nodes or optimizing query execution plans (e.g., using Indexes or Caching). * If

starrocks_fe_query_queue_pending

is high: You are hitting concurrency limits. If CPU/Memory usage is low, you can increase

query_queue_concurrency_limit

or adjust your Resource Group

concurrency_limit

. * Enable Pipeline Engine: Ensure the Pipeline Engine is enabled (default in recent versions) as it handles high concurrency much more efficiently than the legacy engine' than the old fragment-based engine. References • administration/management/monitoring/metrics.md • administration/management/resource_management/query_queues.md

@Rocky

pipe_driver_queue_len

is not showing as a prometheus metric. what is casusing these slowdowns: in my query profile: ScheduleTime │ Fragment 1 MAX 14.923s QueryPeakScheduleTime │ 14.923s PendingTime → InputEmptyTime │ Fragment 0 13.573s EXCHANGE_SINK WaitTime │ MAX 10.018s

The metrics you've provided indicate that your compute nodes (BE/CN) are suffering from extreme CPU saturation and scheduling thrashing. When

ScheduleTime

is high, the system is unable to process the number of tasks being thrown at it. Here is a breakdown of what those specific profile metrics mean for your slowdown: 1.

ScheduleTime

(14.9s) — The "Smoking Gun" This is the most critical metric. It represents the time a query task (Pipeline Driver) spent in the Ready Queue waiting for an available CPU core. * What it means: Your BE nodes are completely overloaded. The tasks are ready to run, but there are no free CPU threads to pick them up. * Target: For a healthy system, this should be in the millisecond range. 14.9s means the CPU is likely context-switching excessively or simply has way too much work. 2.

EXCHANGE_SINK WaitTime

(10.0s) — Backpressure This measures how long a "Sender" node waited because the "Receiver" node's input buffer was full. * What it means: The receiver is not consuming data fast enough. Since your

ScheduleTime

is also high, the receiver is likely slow because its own CPU tasks are stuck in the same scheduling queue. This creates a "logjam" across the cluster. 3.

PendingTime → InputEmptyTime

(13.5s) — Downstream Starvation This is the downstream fragment waiting for data to arrive from an upstream fragment. * What it means: This is a symptom, not the cause. It's high because the upstream fragment is stuck in

ScheduleTime

(waiting for CPU) or

EXCHANGE_SINK

(waiting for network/buffers). ────────── How to support higher QPS To handle higher QPS, you need to reduce the "weight" of each individual query so the CPU can switch between them more efficiently. 1. Reduce `pipeline_dop`: By default, StarRocks tries to use a high number of cores for a single query. For high QPS workloads, this is counter-productive because it creates too many small tasks that fight for CPU. * Action: Try setting

SET GLOBAL pipeline_dop = 4;

(or even

1

or

2

for very high QPS). This limits how many cores a single query can grab, leaving more room for other concurrent queries. 2. Find the Missing Metric: If

pipe_driver_queue_len

is missing in Prometheus, it is likely prefixed. Check for: *

starrocks_be_pipe_driver_queue_len

* You can also verify it directly on a BE node by running:

curl http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep pipe_driver_queue_len

. 3. Check for Data Skew: If

EXCHANGE_SINK WaitTime

is only high on one or two specific nodes, you likely have data skew. One node is receiving way more data than others and becoming the bottleneck for the entire cluster. 4. Monitor `starrocks_be_cpu_utilization`: If this is consistently above 80-90% during these slowdowns, you simply need more compute resources (scale out your CN/BE nodes) to support your target QPS. Summary Recommendation: Your system is currently "thrashing." Start by lowering

pipeline_dop

to 4 or 2 globally. This should immediately lower the

ScheduleTime

and improve overall system stability under load under high concurrency. References • knowledge/usage/best-practice.md • sql-reference/System_variable.md • faq/Sql_faq.md • best_practices/query_tuning/query_profile_operator_metrics.md • faq/resource_isolation_faq.md

@Rocky how could this be affecting my system and qps:

These metrics describe a system in a state of concurrency collapse (or thrashing). Here is how this specifically impacts your system and your ability to scale QPS: 1. The "CPU Queue" Bottleneck (

ScheduleTime

= 14.9s)

ScheduleTime

is the time a task spent in the "Ready Queue" waiting for a CPU thread to become available. * System Impact: Your BE/CN nodes have more "active" tasks (drivers) than physical CPU cores. Even if you have 64 cores, if the system is trying to run 500 tasks, each task spends 14.9 seconds just waiting for its "turn" on the CPU. * Effect on QPS: This creates a hard ceiling on QPS. Because each query takes 15s+ to execute, the number of queries you can complete per second is severely limited. High QPS requires queries to finish in milliseconds so that the CPU can be recycled for the next request. 2. The "Logjam" Effect (

EXCHANGE_SINK WaitTime

= 10.0s)

EXCHANGE_SINK WaitTime

happens when a sender node cannot send data because the receiver node's buffer is full. * System Impact: Because the receiver is stuck in the CPU queue (high

ScheduleTime

), it isn't processing the data sitting in its mailbox. This causes "backpressure" that travels upstream. * Effect on QPS: One slow/overloaded node can effectively stall the entire cluster. Even if a node has free CPU, it might sit idle waiting for a congested node to accept its data. 3. The "Starvation" Effect (

InputEmptyTime

= 13.5s) This is the downstream result of the first two issues. Fragment 0 is idle for 13.5s because the data it needs hasn't arrived. * System Impact: Your resources are being used inefficiently. You have memory and threads allocated to Fragment 0, but they are doing nothing except waiting for Fragment 1 to get through its CPU queue. ────────── How this kills QPS When you try to increase QPS in this state, the following happens: 1. Latency Death Spiral: As you add more queries, the number of tasks in the CPU queue increases. This makes

ScheduleTime

even longer. Eventually, queries start timing out before they even get 10% of their required CPU time. 2. Context Switching Overhead: The CPU spends a significant portion of its power just switching between thousands of waiting tasks rather than doing actual calculation. 3. Connection Exhaustion: Queries stay "active" for 20-30 seconds. New incoming queries will soon hit the

max_concurrency

limit or exhaust the FE's connection pool, leading to

Connection Refused

errors. Immediate Fixes for QPS To support higher QPS, you must reduce the number of concurrent tasks per query to prevent the CPU from thrashing: * Reduce Parallelism: Set

SET GLOBAL pipeline_dop = 8;

(or lower). If this is currently set to 0 (auto) or a very high number, StarRocks is over-parallelizing each query, which is great for one big query but terrible for high QPS. * Check Resource Groups: If you have a specific high-concurrency workload, put it in a Resource Group with a hard

concurrency_limit

. It is better to have 50 queries queueing in the FE (pending) than to have 500 queries half-started and fighting for CPU in the BE. * Short-term Scaling: If your CPU usage is consistently >80% and these metrics stay high, you have outgrown your current compute capacity and need to scale out (add more CN/BE nodes). References • best_practices/query_tuning/query_profile_operator_metrics.md • administration/management/resource_management/resource_group.md • sql-reference/System_variable.md -variable) References • best_practices/query_tuning/query_profile_operator_metrics.md

@Rocky how could these settings affect qps: enable_adaptive_sink_dop true enable_runtime_adaptive_dop true max_pipeline_dop 64 pipeline_dop 0 pipeline_sink_dop 0

These settings determine how "wide" each query runs (how many CPU cores it tries to use). In your current state of high

ScheduleTime

, these settings are likely contributing to the bottleneck rather than helping. Here is how each affects your QPS: 1.

pipeline_dop = 0

(The Multiplier Effect) * What it does: Setting this to

0

tells StarRocks to automatically calculate the Degree of Parallelism (DOP). Usually, this defaults to half the number of physical CPU cores on your machine. * Effect on QPS: If you have a 64-core machine, each query will try to use 32 cores. If you run just 4 queries simultaneously, you are already trying to use 128 "threads" on 64 physical cores. * Result: This leads to the 14.9s ScheduleTime you are seeing. The system is over-parallelizing each query, causing tasks to wait in line for a CPU. For high QPS, you should manually set this to a lower value (e.g., 1, 2, or 4). 2.

max_pipeline_dop = 64

(The Ceiling) * What it does: This is the absolute upper limit for the "auto" calculation. * Effect on QPS: Since yours is set to 64, StarRocks feels free to scale a single query up to 64-way parallelism if it thinks the machine can handle it. * Result: In a high-concurrency environment, a ceiling of 64 is way too high. It allows a few complex queries to "steal" all CPU slots, causing other queries to queue up and increasing overall latency. 3.

enable_runtime_adaptive_dop = true

* What it does: A newer feature (v3.2+) that allows the engine to adjust parallelism while the query is running based on system load. * Effect on QPS: While this is designed to help, it cannot overcome a system that is already 100% saturated. If the initial "auto" DOP is too high, the query has already created the "logjam" before the adaptive logic can scale it down. * Result: It's a safety net, but manually lowering

pipeline_dop

is much more effective for stabilizing QPS. 4.

enable_adaptive_sink_dop

&

pipeline_sink_dop = 0

* What it does: These specifically affect Data Loading (INSERT/Broker Load). * Effect on QPS: If you are running heavy loads at the same time as queries, these settings allow the "Sink" (the part that writes data to disk) to use many cores. * Result: This can cause the

EXCHANGE_SINK WaitTime

you saw. If the loading process is grabbing all the CPU/IO resources to sink data, your SELECT queries will stall waiting for their turn. ────────── Recommended Adjustment for Higher QPS If your goal is higher QPS and you are seeing 15s

ScheduleTime

, your system is "over-subscribed." Change your settings to restrict each query so that more queries can run at once:

sql
-- Reduce the "width" of each query to leave room for others
SET GLOBAL pipeline_dop = 4; 

-- Cap the maximum possible parallelism
SET GLOBAL max_pipeline_dop = 8;

Why this helps QPS: By reducing

pipeline_dop

from "Auto (32+)" to

4

, you reduce the number of tasks the CPU has to manage by 8x. This drastically reduces context switching and

ScheduleTime

, allowing the system to complete many small queries quickly instead of a few queries slowly. References • sql-reference/System_variable.md • knowledge/usage/best-practice.md • administration/management/resource_management/resource_group.md • faq/Sql_faq.md • best_practices/query_tuning/query_profile_operator_metrics.md References • sql-reference/System_variable.md • knowledge/usage/best-practice.md

@Rocky how to check if pipeline_dop changes are working

To verify that your

pipeline_dop

changes are taking effect and actually reducing the load on your system, you can use the following three methods: 1. Check the Query Profile (Most Reliable) The most accurate way to see the actual parallelism used during execution is to look at the query profile. 1. Run a query with the profile enabled:

sql
    SET enable_profile = true;
    SET pipeline_dop = 4; -- Set your desired DOP
    SELECT ... ; -- Your query here

2. Get the Profile ID: Run

SHOW PROFILELIST;

and find the

QueryId

for your statement. 3. Analyze the Profile: Run

ANALYZE PROFILE FROM 'your_query_id';

4. Look for `DegreeOfParallelism`: In the output, look under the Pipeline or Fragment sections. You should see: *

DegreeOfParallelism

: This should match the value you set (e.g.,

4

). * If it was previously high (e.g., 32 or 64) and is now 4, the change is working. 2. Verify Session/Global Variables You can verify the configuration is set correctly in your current session:

sql
-- Check current session value
SHOW VARIABLES LIKE 'pipeline_dop';

-- Check global value (if you set it globally)
SHOW GLOBAL VARIABLES LIKE 'pipeline_dop';

3. Monitor the "Success" Metrics If the change is working, you should see an immediate improvement in the metrics you originally reported: *

ScheduleTime

: This should drop significantly (from seconds down to milliseconds) because there are fewer tasks fighting for CPU. *

starrocks_be_pipe_driver_queue_len

: (If you can find it in your Prometheus/Metrics endpoint). This value should decrease as the number of concurrent tasks waiting for a CPU core drops. * CPU Utilization: While CPU usage might still be high, the "Involuntary Context Switches" (visible in some OS-level tools or BE metrics) should decrease, meaning the CPU is doing more real work and less "management." Why you might not see a change: *

pipeline_dop = 0

: If it is still set to

0

, the system is in "adaptive" mode and will likely still choose a high number based on your core count. Ensure you set it to a fixed integer > 0 to force the limit. * Resource Groups: If you are using Resource Groups, the group's own

concurrency_limit

or

cpu_core_limit

might override or complement the

pipeline_dop

setting. Pro-Tip: If you want to test the effect immediately without changing global settings, use a hint in your SQL:

sql
SELECT /*+ SET_VAR(pipeline_dop = 2) */ * FROM table ...

References • best_practices/query_tuning/query_profile_operator_metrics.md • sql-reference/System_variable.md • knowledge/usage/best-practice.md • faq/Sql_faq.md • administration/management/monitoring/metrics.md monitoring/metrics) References • sql-reference/System_variable.md • best_practices/query_tuning/query_profile_operator_metrics.md • knowledge/usage/best-practice.md • faq/Sql_faq.md • administration/management/monitoring/metrics.md

@Rocky would adding additional compute nodes support massive simple scans and higher qps

Yes, adding compute nodes is the most effective way to address the CPU queue bottleneck (

ScheduleTime

= 14.9s) you reported. Here is how adding nodes helps your specific scenario: 1. Scaling for Higher QPS (Concurrency) Your current metrics show that your BE/CN nodes have more tasks than they have CPU threads to handle them. * Total "Slots": If one node has 32 cores, it can effectively handle a certain number of concurrent tasks. By adding 5 more nodes, you increase your cluster's total "thread pool" by 160 cores. * Reduced Queuing: With more nodes, incoming queries are distributed across a larger pool of resources. This directly reduces the

ScheduleTime

because tasks spend less time waiting in the "Ready Queue" for a free CPU core. 2. Scaling for Massive Simple Scans (Throughput) Simple scans are often I/O and CPU-bound. Adding nodes helps in two ways: * More Scanner Threads: Each node has its own

scanner_thread_pool

. Adding nodes increases the total number of threads reading from disk/object storage simultaneously across the cluster. * I/O Parallelism: * Shared-Nothing (BE): Adding BE nodes and rebalancing data spreads your data across more physical disks. This allows you to read from 20 disks instead of 5, dramatically increasing total scan throughput. * Shared-Data (CN): Adding Compute Nodes allows more machines to pull data from S3/HDFS in parallel. Since CNs are stateless, they can scale out in seconds to handle a sudden burst in scan volume. 3. Which Node Type Should You Add? * Add BEs (Shared-Nothing): Use this if your data is stored locally on StarRocks. Note that after adding BEs, you should ensure data is rebalanced (StarRocks does this automatically over time) so the new nodes actually host data to scan. * Add CNs (Shared-Data): Use this if you are using StarRocks in Shared-Data mode or scanning External Tables (Hive, Iceberg, Hudi). CNs are "stateless" and can be added or removed instantly without moving data. When Adding Nodes Won't Help While adding nodes helps with CPU and I/O, it won't fix these specific bottlenecks: 1. FE Bottleneck: If your FE CPU is at 100% due to query parsing/planning, adding BE/CN nodes won't help. You would need to add Observer FE nodes to handle the high QPS of "simple" queries. 2. Single Tablet Limit: If your table only has 1 tablet (bucket), only 1 BE can scan it. Adding 100 nodes won't make that specific scan faster. Ensure your

DISTRIBUTED BY

clause has enough buckets to take advantage of the new nodes. Recommendation If your

ScheduleTime

remains high even after you reduce

pipeline_dop

, your cluster is simply under-provisioned for your current load. Adding nodes is the correct path to restore millisecond-level responsiveness. References • introduction/Architecture.md • deployment/plan_cluster.md • administration/management/BE_configuration.md • administration/management/resource_management/resource_group.md • faq/Deploy_faq.md References • administration/management/BE_configuration.md • faq/Deploy_faq.md