< Rocky> what are the pros and cons of Starrocks query queue StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> what are the pros and cons of Starr...

Philip Oliva

03/09/2026, 5:10 PM

@Rocky what are the pros and cons of Starrocks query queues? https://docs.starrocks.io/docs/administration/management/resource_management/query_queues/ Why is it disabled by default? Does that mean its preferred to be disabled?

Rocky

03/09/2026, 5:10 PM

The query queue feature in StarRocks is a resource management tool designed to protect the system from being overwhelmed during high-concurrency scenarios or when processing resource-intensive queries. Here is a breakdown of the pros and cons, along with an explanation of why it is disabled by default. Pros of Query Queues * System Stability: Prevents "cascading failures" where too many concurrent queries lead to Out of Memory (OOM) crashes or CPU starvation, which can take down an entire Backend (BE) node. * Resource Fairness: By setting thresholds at the Resource Group level (available from v3.1.4+), you can ensure that one user or application doesn't hog all the cluster resources, leaving some for others. * Dynamic Concurrency (v3.1.4+): StarRocks can dynamically adjust the

pipeline_dop

(degree of parallelism) for incoming queries based on current load, optimizing throughput without manual tuning. * Predictable Performance: It cushions the system against sudden "bursty" workloads. Instead of all queries slowing down simultaneously to a crawl, some are queued while others finish at full speed. * Graceful Handling: You can set a maximum queue size and timeout, so if the system is truly at capacity, it can reject new queries with a clear error rather than hanging. Cons of Query Queues * Increased Latency: The most obvious downside is that queries in the queue will have a higher "wall clock" time. A query that takes 1 second to run might wait 5 seconds in the queue. * Configuration Overhead: There is no "one-size-fits-all" threshold. You have to manually tune the

query_queue_concurrency_limit

query_queue_mem_used_pct_limit

, and CPU limits based on your specific hardware and workload. * Monitoring Complexity: Once enabled, you need to monitor queue lengths and wait times to ensure your thresholds aren't too restrictive (causing idle resources) or too loose (failing to protect the system). ────────── Why is it disabled by default? Query queues are disabled by default for several strategic reasons: 1. Workload Diversity: Every environment is different. A threshold that works for a 1TB RAM cluster would be catastrophic for a 16GB RAM development machine. StarRocks leaves it off so that users aren't met with "Query Queued" errors immediately upon installation before they've had a chance to benchmark their hardware. 2. Maximum Out-of-the-Box Throughput: In many analytics use cases, users want to see the maximum "raw" power of the engine. Queuing introduces an artificial ceiling that might hide the true performance of the cluster during initial testing. 3. Potential for Idle Resources: If thresholds are set too conservatively, the system might queue queries even when the CPU and Memory have plenty of headroom, leading to under-utilization of expensive hardware. Does "disabled by default" mean it's preferred to be disabled? No. In a production environment, it is generally recommended to enable and configure query queues, especially if: * You have a high number of concurrent users. * You have mixed workloads (e.g., fast dashboard queries running alongside heavy ETL or report exports). * You want to prevent accidental OOMs caused by a sudden spike in complex queries. Summary Recommendation: * Disable it during: Initial benchmarking or if you have a very controlled, low-concurrency environment. * Enable it during: Production deployments with multi-tenant access or high concurrency to ensure "noisy neighbors" don't crash the cluster. To enable it for SELECT queries, you can use:

Copy code

sql
SET GLOBAL enable_query_queue_select = true;
-- Then set your specific limits, e.g., max 100 concurrent queries per BE
SET GLOBAL query_queue_concurrency_limit = 100;

References • administration/management/resource_management/query_queues.md

Rocky

03/09/2026, 5:10 PM

• knowledge/usage/best-practice.md • administration/management/monitor_manage_big_queries.md

Philip Oliva

03/09/2026, 5:29 PM

Philip Oliva

03/09/2026, 5:29 PM

@Rocky Specifically why would I enable

enable_group_level_query_queue

vs not if I want queues?

Rocky

03/09/2026, 5:29 PM

The choice to enable

enable_group_level_query_queue

depends on whether you want a single "global" traffic cop for your cluster or individual "per-workload" traffic cops. Here is the specific breakdown of why you would use one versus the other: 1. Global Queue Only (
enable_group_level_query_queue = false
) If this is off, StarRocks only looks at cluster-wide thresholds (like

query_queue_concurrency_limit

). * Why choose this: You have a simple workload where you just want to prevent the BE nodes from crashing due to overload. You don't care who is running the queries; if the system is busy, everyone waits in the same line. * The Downside: A single "noisy neighbor" (e.g., a user running 100 heavy analytical queries) can fill up the global queue, causing a high-priority dashboard query from another user to get stuck behind them. 2. Group-Level Queue (
enable_group_level_query_queue = true
) Introduced in v3.1.4, this allows each Resource Group to have its own queue and limits (

concurrency_limit

max_cpu_cores

). * Why choose this (The "Why"): * Isolation & SLAs: You can ensure that your "ETL Group" is limited to 5 concurrent queries and queues up when it hits that limit, while your "Data Science Group" has its own limit. One group hitting its limit will not force the other group to queue unless the global limit is also hit. * Prioritization: You can give a "VIP Group" a very high concurrency limit so they almost never queue, while limiting "Ad-hoc" users to a small number of slots. * Multi-tenancy: If you are hosting multiple departments on one cluster, this is the only way to prevent one department from starving the others of query slots. Key Behavioral Differences Feature | Global Queue Only | Group-Level Queue Enabled Trigger Logic | Based on total BE load/concurrency. | Based on both Resource Group limits AND Global limits. Queuing Policy | First-in, First-out (FIFO) globally. | FIFO within the specific resource group. Resource Control | Protects the BE from crashing. | Protects workloads from each other (Isolation). Granularity | Coarse (All or nothing). | Fine-grained (Per user/workload group). Summary Recommendation * Enable it if: You use Resource Groups to separate different types of traffic (e.g., Loading vs. Ad-hoc vs. Dashboard) and want to ensure that a surge in one type of traffic doesn't block the others. * Leave it disabled if: You have a very uniform workload and your only goal is a "safety valve" to prevent the cluster from being overwhelmed by total volume. Note: To use group-level queuing, you must also have the global switches enabled (e.g.,

SET GLOBAL enable_query_queue_select = true;

) and have defined

concurrency_limit

in your

CREATE RESOURCE GROUP

statements. References • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md

Philip Oliva

03/09/2026, 10:03 PM

@Rocky if your scaling CNs at 70% cpu are these good settings for queuing queries? `SET GLOBAL enable_query`_`queue_load = true;`

SET GLOBAL enable_query_queue_select = true;

SET GLOBAL enable_query_queue_statistic = true;

SET GLOBAL query_queue_cpu_used_permille_limit = 800;

SET GLOBAL query_queue_max_queued_queries = 0;

SET GLOBAL query_queue_pending_timeout_second = 600;

Rocky

03/09/2026, 10:03 PM

The settings you've outlined are generally a solid foundation for protecting your cluster while scaling. Since you are triggering scaling at 70% CPU, setting the query queue to kick in at 80% (800 permille) creates a 10% safety buffer. This allows the cluster to handle short bursts naturally while ensuring that if scaling doesn't happen fast enough, the system won't grind to a halt. Here is a breakdown of your settings with some specific recommendations: 1. CPU Threshold (
query_queue_cpu_used_permille_limit = 800
) * Verdict: Good choice. * Reasoning: Since your autoscaler triggers at 70%, setting the queue at 80% is logical. It prevents the CPU from hitting 90-100%, which usually leads to increased latency, RPC timeouts, and potential node instability. * Tip: If you find that performance degrades significantly before hitting 80%, or if your queries are very sensitive to CPU steal, you might want to tighten this to 750 (75%) to be closer to your scaling trigger. 2. Max Queued Queries (
query_queue_max_queued_queries = 0
) * Verdict: Risky. * Reasoning: In StarRocks, setting this to

means unlimited. While this prevents queries from being rejected immediately, it can lead to: * Memory Pressure on FE: Every queued query consumes memory on the Leader FE. An unlimited queue during a massive spike could lead to FE OOM (Out of Memory). * "Zombies" in the Queue: If your scaling takes 5 minutes and you have an unlimited queue, you might end up with thousands of stale queries that users have already abandoned. * Recommendation: Set this to a reasonable limit like 1024 or 2048. If your queue exceeds 2000 queries, it’s usually better to fail the query so the application can retry or the user knows the system is busy. 3. Timeout (
query_queue_pending_timeout_second = 600
) * Verdict: A bit high. * Reasoning: 10 minutes is a long time for a query to sit in a queue. Most interactive users or BI tools will have timed out on the client side long before then. * Recommendation: If your scaling typically completes within 3-5 minutes, a timeout of 300 (5 minutes) is usually safer. It clears out the queue faster if the system is genuinely stuck. 4. Statistic Queries (
enable_query_queue_statistic = true
) * Verdict: Excellent. * Reasoning: Background

ANALYZE

jobs can be surprisingly CPU-intensive. Queuing them alongside user queries ensures they don't interfere with production workloads during a high-load event. Summary of Recommended Adjustments If you want a more "production-safe" version of your configuration, consider these tweaks:

Copy code

sql
-- Protect FE memory by setting a limit instead of 0 (unlimited)
SET GLOBAL query_queue_max_queued_queries = 1024; 

-- 5 minutes is usually enough for scaling to catch up
SET GLOBAL query_queue_pending_timeout_second = 300; 

-- Keep your existing choices for these
SET GLOBAL enable_query_queue_load = true;
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL enable_query_queue_statistic = true;
SET GLOBAL query_queue_cpu_used_permille_limit = 800;

One final tip: If you are on StarRocks v3.1.4 or higher, ensure you are using the Leader FE to manage these queues. StarRocks in these versions uses "Collective Concurrency Management," which is much more accurate for managing load across multiple CNs than earlier versions. References • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • administration/management/monitor_manage_big_queries.md

Philip Oliva

03/10/2026, 1:47 PM

@Rocky What is a the safe way to scale down a CN without FEs hitting errors.

Rocky

03/10/2026, 1:47 PM

To safely scale down a Compute Node (CN) in StarRocks without causing query errors on the Frontend (FE), you should use the Graceful Exit mechanism combined with the

DROP

command. Unlike Backend (BE) nodes, CNs are stateless and cannot be decommissioned using

ALTER SYSTEM DECOMMISSION

. Instead, follow these steps to ensure in-flight queries complete before the node is removed: 1. Trigger Graceful Exit on the CN Instead of simply killing the process or dropping it from the FE immediately, go to the CN node and run the stop script with the

-g

(graceful) flag:

Copy code

bash
# Navigate to your CN deployment directory
cd <cn_dir>/be
./bin/stop_cn.sh -g

What happens: * The CN marks itself as "exiting" and informs the FE. * The FE stops sending new query fragments to this CN. * The CN continues to process existing in-flight query fragments. * The CN will wait for a duration (default 20 seconds, controlled by

loop_count_wait_fragments_finish

cn.conf

) for queries to finish before shutting down the process. 2. Verify the CN Status Check the status of the CN from your SQL client to ensure it has stopped:

Copy code

sql
SHOW COMPUTE NODES;

Look for the

Alive

column (should be

false

) and the

StatusCode

(should show

SHUTDOWN

DISCONNECTED

). 3. Permanently Remove the CN Once the CN process has safely stopped and in-flight queries have finished, remove the node from the cluster metadata:

Copy code

sql
ALTER SYSTEM DROP COMPUTE NODE "cn_host:cn_heartbeat_service_port";

Summary of Best Practices * Version Note: Graceful Exit is supported from v3.3 onwards. From v3.4, the FE is even more "aware" of the exiting state, which further reduces the risk of query failures during this window. * Do not use `DECOMMISSION`: The

ALTER SYSTEM DECOMMISSION

command is specifically for BE nodes to migrate data replicas. Since CNs do not store data, this command will return an error or not work for them. * Wait for Process Exit: Always ensure the

stop_cn.sh -g

command has completed its wait loop before executing the

DROP

command to ensure the FE has completely redirected all traffic. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • deployment/downgrade.md • administration/management/Scale_up_down.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • deployment/upgrade.md

Philip Oliva

03/10/2026, 1:50 PM

Philip Oliva

03/10/2026, 1:51 PM

@Rocky If you using kube hpa to scale in and out how do I exit CNs gracefully?

Rocky

03/10/2026, 1:51 PM

To gracefully exit Compute Nodes (CNs) during a Kubernetes HPA scale-in event, you need to coordinate StarRocks' internal Graceful Exit mechanism with Kubernetes' Pod lifecycle. From StarRocks v3.4 onwards, Graceful Exit is fully supported and enabled by default, making it the recommended version for production HPA setups. 1. Key StarRocks Configurations Ensure these parameters are set in your CN configuration (

cn.conf

) or passed as environment variables in your Kubernetes manifest: * loop_count_wait_fragments_finish
: This controls how long the CN waits for in-flight query fragments to complete. * Calculation:

Value * 10 seconds

. * Default:

(20 seconds). * Recommendation: Set this based on your longest expected "short-to-medium" queries (e.g.,

for 60 seconds). * graceful_exit_wait_for_frontend_heartbeat
(Available from v3.4.5): * Set to

true

. This ensures the CN waits for the FE to acknowledge its "Exiting" status via heartbeat, which prevents the FE from marking the node as "Dead" prematurely and allows it to reroute new traffic away from the scaling node. 2. Kubernetes Pod Configuration When HPA triggers a scale-in, Kubernetes sends a

SIGTERM

signal to the CN process. StarRocks CNs are designed to catch

SIGTERM

and initiate the Graceful Exit sequence. To ensure this completes successfully, you must adjust the terminationGracePeriodSeconds
in your StarRocksCluster CRD or Pod spec: * terminationGracePeriodSeconds
: This must be greater than the StarRocks wait time. * Formula:

(loop_count_wait_fragments_finish * 10) + buffer (e.g., 20s)

. * Example: If

loop_count_wait_fragments_finish

(60s), set

terminationGracePeriodSeconds

to at least

. 3. Graceful Exit Process Flow When Kube HPA scales down: 1. Kubernetes sends

SIGTERM

to the CN Pod. 2. CN receives

SIGTERM

, marks itself as Exiting, and stops accepting new query fragments (returns

INTERNAL_ERROR

for new requests). 3. FE (in v3.4+) detects the "Exiting" state and stops scheduling new tasks to this node. 4. CN continues processing existing query fragments until they finish or the

loop_count_wait_fragments_finish

timeout is reached. 5. CN shuts down once fragments are cleared. 6. Kubernetes force-kills the container only if the process exceeds the

terminationGracePeriodSeconds

loop_count_wait_fragments_finish

| Set to cover your average query duration FE Awareness |

graceful_exit_wait_for_frontend_heartbeat

true

K8s Grace Period |

terminationGracePeriodSeconds

(CN Wait Time) + 20s

Note: If you are using the StarRocks Operator, you can define these under the

starRocksCnSpec

in your

StarRocksCluster

YAML. If your CNs are not exiting gracefully, check the

cn.log

to see if the process is being killed by Kubernetes before it finishes its loop. References • administration/management/graceful_exit.md • administration/stargo.md

Philip Oliva

04/10/2026, 12:38 AM

@Rocky Is there explanation for Deadline Exceeded errors in fe.log? 2026-04-09 191810.358-04:00 WARN (thrift-server-pool-423|4800) [DefaultCoordinator.updateRuntimeProfile():1195] exec state report failed status=errorCode INTERNAL_ERROR starlet err grpc.GetShard(shardId=11260) error: Deadline Exceeded: BE:107114, query_id=5abb0672-346a-11f1-8c10-56e947f44903, instance_id=5abb0672-346a-11f1-8c10-56e947f44908, backend_id=107114 2026-04-09 191810.359-04:00 INFO (starrocks-mysql-nio-pool-111|3108) [BackendResourceStat.getAvgNumHardwareCoresOfBe():141] update avgNumHardwareCoresOfBe to 62, current cpuCores stats: {10001=62, 94451=62, 94450=62, 107093=62, 107156=62, 107114=62, 107135=62} 2026-04-09 191810.360-04:00 INFO (starrocks-mysql-nio-pool-90|2899) [BackendResourceStat.getAvgNumHardwareCoresOfBe():141] update avgNumHardwareCoresOfBe to 62, current cpuCores stats: {10001=62, 94451=62, 94450=62, 107093=62, 107156=62, 107114=62, 107135=62} 2026-04-09 191810.361-04:00 INFO (starrocks-mysql-nio-pool-186|6318) [BackendResourceStat.getAvgNumHardwareCoresOfBe():141] update avgNumHardwareCoresOfBe to 62, current cpuCores stats: {10001=62, 94451=62, 94450=62, 107093=62, 107156=62, 107114=62, 107135=62} 2026-04-09 191810.362-04:00 WARN (starrocks-mysql-nio-pool-59|2780) [DefaultCoordinator.getNext():950] get next fail, need cancel. status errorCode CANCELLED starlet err grpc.GetShard(shardId=11260) error: Deadline Exceeded: BE:107114, query id: 5abb0672-346a-11f1-8c10-56e947f44903 2026-04-09 191810.362-04:00 WARN (starrocks-mysql-nio-pool-59|2780) [DefaultCoordinator.dealStatusToTryRetry():913] query 5abb0672-346a-11f1-8c10-56e947f44903 failed: starlet err grpc.GetShard(shardId=11260) error: Deadline Exceeded: BE:107114 2026-04-09 191810.362-04:00 WARN (starrocks-mysql-nio-pool-59|2780) [StmtExecutor.execute():854] Query 5abb0672-346a-11f1-8c10-56e947f44903 failed. Planner profile : Planner: 2026-04-09 191810.362-04:00 INFO (starrocks-mysql-nio-pool-106|3071) [BackendResourceStat.getAvgNumHardwareCoresOfBe():141] update avgNumHardwareCoresOfBe to 62, current cpuCores stats: {10001=62, 94451=62, 94450=62, 107093=62, 107156=62, 107114=62, 107135=62} 2026-04-09 191810.363-04:00 INFO (starrocks-mysql-nio-pool-59|2780) [StmtExecutor.execute():954] execute Exception, sql: /* * 2026-04-09 191810.365-04:00 WARN (thrift-server-pool-423|4800) [DefaultCoordinator.updateRuntimeProfile():1195] exec state report failed status=errorCode INTERNAL_ERROR starlet err grpc.GetShard(shardId=13389) error: Deadline Exceeded: BE:107114, query_id=5aaad9bf-346a-11f1-8c10-56e947f44903, instance_id=5aaad9bf-346a-11f1-8c10-56e947f44905, backend_id=107114 2026-04-09 191810.365-04:00 WARN (thrift-server-pool-423|4800) [DefaultCoordinator.updateStatus():885] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: 5aaad9bf-346a-11f1-8c10-56e947f44903, instance id: 5aaad9bf-346a-11f1-8c10-56e947f44905 2026-04-09 191810.365-04:00 INFO (thrift-server-pool-423|4800) [DefaultCoordinator.cancelInternal():1049] query: 5aaad9bf-346a-11f1-8c10-56e947f44903 has unfinished instances: [5aaad9bf-346a-11f1-8c10-56e947f44910, 5aaad9bf-346a-11f1-8c10-56e947f44912, 5aaad9bf-346a-11f1-8c10-56e947f44911, 5aaad9bf-346a-11f1-8c10-56e947f44904, 5aaad9bf-346a-11f1-8c10-56e947f44907, 5aaad9bf-346a-11f1-8c10-56e947f44906, 5aaad9bf-346a-11f1-8c10-56e947f44930, 5aaad9bf-346a-11f1-8c10-56e947f4490c, 5aaad9bf-346a-11f1-8c10-56e947f4490f, 5aaad9bf-346a-11f1-8c10-56e947f4490e, 5aaad9bf-346a-11f1-8c10-56e947f44908, 5aaad9bf-346a-11f1-8c10-56e947f4490b, 5aaad9bf-346a-11f1-8c10-56e947f4490a, 5aaad9bf-346a-11f1-8c10-56e947f4490d, 5aaad9bf-346a-11f1-8c10-56e947f44909, 5aaad9bf-346a-11f1-8c10-56e947f4492c, 5aaad9bf-346a-11f1-8c10-56e947f4492f, 5aaad9bf-346a-11f1-8c10-56e947f4492e, 5aaad9bf-346a-11f1-8c10-56e947f4492b, 5aaad9bf-346a-11f1-8c10-56e947f4492a, 5aaad9bf-346a-11f1-8c10-56e947f4492d] 2026-04-09 191810.365-04:00 WARN (thrift-server-pool-422|4793) [DefaultCoordinator.updateRuntimeProfile():1195] exec state report failed status=errorCode INTERNAL_ERROR starlet err grpc.GetShard(shardId=13389) error: Deadline Exceeded: BE:107114, query_id=5aaad9bf-346a-11f1-8c10-56e947f44903, instance_id=5aaad9bf-346a-11f1-8c10-56e947f44935, backend_id=107114 2026-04-09 191810.366-04:00 WARN (starrocks-mysql-nio-pool-28|2725) [DefaultCoordinator.getNext():950] get next fail, need cancel. status errorCode CANCELLED starlet err grpc.GetShard(shardId=13389) error: Deadline Exceeded: BE:107114, query id: 5aaad9bf-346a-11f1-8c10-56e947f44903 2026-04-09 191810.366-04:00 WARN (starrocks-mysql-nio-pool-28|2725) [DefaultCoordinator.dealStatusToTryRetry():913] query 5aaad9bf-346a-11f1-8c10-56e947f44903 failed: starlet err grpc.GetShard(shardId=13389) error: Deadline Exceeded: BE:107114 2026-04-09 191810.366-04:00 WARN (thrift-server-pool-420|4698) [DefaultCoordinator.updateRuntimeProfile():1195] exec state report failed status=errorCode INTERNAL_ERROR starlet err grpc.GetShard(shardId=13389) error: Deadline Exceeded: BE:107114, query_id=5aaad9bf-346a-11f1-8c10-56e947f44903, instance_id=5aaad9bf-346a-11f1-8c10-56e947f4492e, backend_id=107114 2026-04-09 191810.366-04:00 WARN (thrift-server-pool-418|4681) [DefaultCoordinator.updateRuntimeProfile():1195] exec state report failed status=errorCode INTERNAL_ERROR starlet err grpc.GetShard(shardId=13389) error: Deadline Exceeded: BE:107114, query_id=5aaad9bf-346a-11f1-8c10-56e947f44903, instance_id=5aaad9bf-346a-11f1-8c10-56e947f4490f, backend_id=107114 2026-04-09 191810.366-04:00 WARN (starrocks-mysql-nio-pool-28|2725) [StmtExecutor.execute():854] Query 5aaad9bf-346a-11f1-8c10-56e947f44903 failed. Planner profile : Planner: 2026-04-09 191810.366-04:00 INFO (starrocks-mysql-nio-pool-84|2893) [BackendResourceStat.getAvgNumHardwareCoresOfBe():141] update avgNumHardwareCoresOfBe to 62, current cpuCores stats: {10001=62, 94451=62, 94450=62, 107093=62, 107156=62, 107114=62, 107135=62} 2026-04-09 191810.368-04:00 INFO (starrocks-mysql-nio-pool-28|2725) [StmtExecutor.execute():954] execute Exception, sql: /* * 2026-04-09 191810.378-04:00 INFO (starrocks-mysql-nio-pool-127|3289) [BackendResourceStat.getAvgNumHardwareCoresOfBe():141] update avgNumHardwareCoresOfBe to 62, current cpuCores stats: {10001=62, 94451=62, 94450=62, 107093=62, 107156=62, 107114=62, 107135=62} 2026-04-09 191810.378-04:00 INFO (starrocks-mysql-nio-pool-42|2760) [BackendResourceStat.getAvgNumHardwareCoresOfBe():1

Rocky

04/10/2026, 12:38 AM

The "Deadline Exceeded" error in your

fe.log

indicates that a gRPC request from the Frontend (FE) to a Backend (BE/CN) timed out. Specifically, the error

starlet err grpc.GetShard

identifies this as a Shared-data (Cloud-native) cluster issue involving the Starlet service. 1. What is happening? In a shared-data architecture, StarRocks uses a component called Starlet (running on BE/CN nodes) to manage tablet/shard metadata and interaction with object storage (S3, GCS, etc.). * GetShard
: This is a gRPC call the FE makes to a BE to retrieve or verify metadata for a specific data shard. * Deadline Exceeded
: This is a standard gRPC error (Code 4) meaning the FE waited for a response but reached its internal timeout (deadline) before the BE replied. 2. Common Causes Based on your logs, where multiple queries failed on the same backend (

BE:107114

) at the exact same time (

19:18:10

), the issue is likely node-specific: * Backend Overload: The BE node

may have been experiencing a spike in CPU usage, memory pressure (causing long GC pauses), or extreme Disk I/O. This prevents the Starlet service from responding to gRPC requests within the time limit. * Object Storage Latency: If the BE's

GetShard

operation requires fetching metadata from your object storage (e.g., S3), and the object storage is experiencing high latency or throttling, the gRPC call will time out. * Network Congestion: Temporary network instability between the FE and that specific BE node. * Timeout Settings: The default timeout for these metadata operations might be too low for your current workload or environment. 3. Troubleshooting & Resolution Step 1: Inspect Backend
107114
Since the logs consistently point to

BE:107114

, focus your investigation there: * Check
<http://be.INFO|be.INFO>
logs: Look for logs around

2026-04-09 19:18:10

. Search for keywords like

starlet

GetShard

, or

slow

. * Monitor Resources: Check if the node had a CPU spike or high Load Average at that time. * Check for GC: Look for "Long GC pause" messages in the BE logs or monitor the process's memory usage. Step 2: Check Object Storage Health Confirm there were no service interruptions or performance degradations with your object storage provider (S3, OSS, GCS, etc.) during that window. Step 3: Adjust FE Configuration (If necessary) If your environment has inherently high latency (e.g., cross-region object storage) or very high metadata volume, you can try increasing the gRPC timeout on the FE: * starmgr_grpc_timeout_seconds
: This parameter controls the timeout for Starlet-related gRPC calls. The default is usually 5 seconds. * To check:

ADMIN SHOW CONFIG LIKE 'starmgr_grpc_timeout_seconds';

* To increase (e.g., to 10s):

ADMIN SET FRONTEND CONFIG ("starmgr_grpc_timeout_seconds" = "10");

* starmgr_grpc_server_max_worker_threads
: If you have a very large number of concurrent queries, the Starlet gRPC server might be running out of threads. (Default: 1024). Step 4: Verify Network Connectivity Ensure that the

starlet_port

(default

) on the BE is not being throttled or blocked by firewalls between the FE and BE. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md

Philip Oliva

04/10/2026, 12:41 AM

@Rocky Is Java GC used on CNs too?

Rocky

04/10/2026, 12:41 AM

Yes, Java Garbage Collection (GC) is used on Compute Nodes (CNs), but it works differently than on the Frontend (FE). While the StarRocks CN/BE core engine is written in C++, it initializes and hosts a Java Virtual Machine (JVM) via JNI (Java Native Interface) to support specific features. Where Java GC occurs on a CN The JVM on a CN is primarily used for: * External Catalogs: Connecting to data sources via JDBC (which requires Java drivers). * Java UDFs: Executing user-defined functions written in Java. * HDFS/Cloud Integration: Interacting with HDFS or certain object storage libraries that rely on the Hadoop client (Java-based). If you are using any of these features, the JVM will perform Garbage Collection. A "Stop-The-World" (STW) GC pause in the CN's JVM can stall the threads calling into Java, potentially contributing to the "Deadline Exceeded" or timeout errors you see in the

fe.log

. How to Monitor and Configure CN Java GC Unlike the FE, which has a dedicated

fe.gc.log

, CN nodes typically do not enable JVM GC logging by default. 1. Configuration: You can control the CN's JVM settings (including heap size and GC strategy) using the

JAVA_OPTS

parameter in your

cn.conf

(or

be.conf

). * Example:

JAVA_OPTS="-Xmx8g -Xms8g -XX:+UseG1GC"

2. Symptoms: If a CN is experiencing heavy Java GC, you might see high CPU usage that doesn't correlate with query execution in the C++ engine, or specific "JNI" or "External Table" queries becoming extremely slow. 3. The "Starlet" Connection: In your specific error (

starlet err grpc.GetShard

), the Starlet component itself is written in C++. However, if the CN node is generally unresponsive due to a severe system-wide resource contention (triggered by a massive JVM GC or OOM), Starlet's gRPC responses will be delayed, leading to that "Deadline Exceeded" error. Recommendation If you suspect Java GC is the culprit: * Check if you are running heavy Java UDFs or JDBC Catalog queries. * Increase the JVM heap size in

cn.conf

via

JAVA_OPTS

if you are seeing memory pressure. * Check the CN node's overall memory usage to ensure the JVM isn't competing for RAM with the StarRocks C++

mem_limit

. References • introduction/StarRocks_intro.md • sql-reference/sql-functions/JAVA_UDF.md • unloading/arrow_flight.md • developers/development-environment/ide-setup.md • introduction/Architecture.md

Philip Oliva

04/10/2026, 12:46 AM

@Rocky https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/charts/starrocks/values.yaml#L769 does not show

JAVA_OPTS

example for CNs. It does only in FE config section https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/charts/starrocks/values.yaml#L396.

Rocky

04/10/2026, 12:46 AM

You are correct that the provided Helm

values.yaml

example for the CN (Compute Node) section is less detailed than the FE section. However, the StarRocks Kubernetes Operator supports setting JAVA_OPTS
for CNs (and BEs) using the cnEnvVars
parameter. Because the StarRocks CN/BE startup scripts check for the

JAVA_OPTS

environment variable to initialize the internal JVM, you can pass it through the Helm chart like this: How to configure
JAVA_OPTS
for CNs In your

values.yaml

(or custom overrides file), add the following under the

starrocksCnSpec

section:

Copy code

yaml
starrocksCnSpec:
  # ... other CN configs ...
  cnEnvVars:
    - name: JAVA_OPTS
      value: "-Xmx8g -Xms8g -XX:+UseG1GC"

Key Details 1. cnEnvVars
: This field is a list of environment variables passed directly to the CN container. Since the CN's entrypoint script (

cn_entrypoint.sh

) inherits the shell's environment, setting

JAVA_OPTS

here will correctly configure the JVM used for Java UDFs, External Catalogs (JDBC/HDFS), and other Java-based integrations. 2. Memory Management: When setting

-Xmx

, ensure it fits within the

resources.limits.memory

you have defined for the CN pod. Typically, you want the JVM heap to be a fraction of the total container memory (e.g., 10-20%) because the C++ engine (the primary component of the CN) manages its own memory via the

mem_limit

configuration. 3. BE Nodes: If you are using dedicated BE nodes instead of CNs, the parameter is identical but named beEnvVars
under the

starRocksBeSpec

section. Why this matters for "Deadline Exceeded" errors As mentioned previously, if the CN's JVM experiences a "Stop-the-World" Garbage Collection pause, it can block the Starlet service or JNI calls, leading to the gRPC timeouts you saw in your logs. Configuring explicit GC settings (like

-XX:+UseG1GC

) and ensuring sufficient heap via

JAVA_OPTS

is the primary way to prevent these Java-related stalls on the Compute Nodes. References • sql-reference/sql-functions/JAVA_UDF.md • deployment/sr_operator.md

Philip Oliva

04/10/2026, 12:56 AM

@Rocky Is there kubectl command on a CN that can do this? I don't know where the java binary would be on stock image. Run this with the same java binary the CN will use ($JAVA_HOME/bin/java on the host where CN runs): java -XX:+PrintFlagsFinal -version 2>&1 | grep -E 'MaxHeapSize|InitialHeapSize'

Rocky

04/10/2026, 12:57 AM

On a stock StarRocks CN image (usually based on Ubuntu or CentOS), the

java

binary is typically already in the system

PATH

or pointed to by the

JAVA_HOME

environment variable. You can use the following

kubectl

command to check the default heap sizes directly: The Recommended Command This command uses the environment variable

$JAVA_HOME

(which StarRocks sets internally) to ensure it uses the exact same binary the CN process uses:

Copy code

bash
kubectl exec <your-cn-pod-name> -n <your-namespace> -- /bin/bash -c '$JAVA_HOME/bin/java -XX:+PrintFlagsFinal -version 2>&1 | grep -E "MaxHeapSize|InitialHeapSize"'

Fallback (if $JAVA_HOME is not set in your shell) If the above returns an error, try using the

java

binary from the system path:

Copy code

bash
kubectl exec <your-cn-pod-name> -n <your-namespace> -- java -XX:+PrintFlagsFinal -version 2>&1 | grep -E "MaxHeapSize|InitialHeapSize"

How to find the Java path manually If you are unsure where Java is located in your specific image, run these two commands: 1. Check the environment variable:

kubectl exec <pod-name> -- printenv JAVA_HOME

2. Locate the binary:

kubectl exec <pod-name> -- which java

Why this matters for your "Deadline Exceeded" issue The output of this command will show you the Default Heap Size the JVM is using. * Stock Behavior: If you haven't set

JAVA_OPTS

in your

cnEnvVars

, the JVM often defaults to 1/4 of the container's physical memory for

MaxHeapSize

. * The Risk: If your CN has 64GB of RAM, the JVM might grab 16GB. If the C++ engine (

mem_limit

) is also trying to use 90% of the RAM, the node will swap or trigger the OOM Killer, causing the gRPC "Deadline Exceeded" errors you observed. Tip: For CNs, it is best practice to keep the Java Heap small (e.g.,

-Xmx4g

-Xmx8g

) unless you are doing very heavy Java UDF or JDBC Catalog processing, to leave maximum room for the C++ engine. References • developers/build-starrocks/Build_in_docker.md • deployment/environment_configurations.md • deployment/prepare_deployment_files.md

Philip Oliva

04/10/2026, 1:26 AM

@Rocky is there tuning params on CNs that would help keep CPU below 70%?

Rocky

04/10/2026, 1:27 AM

Tuning a StarRocks Compute Node (CN) to stay below a specific CPU threshold like 70% involves a combination of Concurrency Control, Resource Isolation, and Query Parallelism settings. Because StarRocks is designed to maximize hardware utilization, it will naturally try to use 100% of the CPU to finish queries as fast as possible. To cap this, you should use the following strategies: 1. Enable Global Query Queuing (Reactive Cap) The most direct way to prevent a node from redlining is to enable the global query queue. This prevents new queries from starting if the node's CPU is already high. Run these SQL commands to set a threshold (e.g., 700 permille = 70%):

Copy code

sql
-- Trigger queuing when CPU usage exceeds 70%
SET GLOBAL query_queue_cpu_used_permille_limit = 700;

-- Optional: Limit max concurrent queries node-wide
SET GLOBAL query_queue_concurrency_limit = 50;

2. Use Resource Groups (Proactive Cap) For more granular control, you can create Resource Groups. Starting in v3.3.5, StarRocks supports "Hard" CPU limits. * Hard Limit (
exclusive_cpu_cores
): Reserves a specific number of cores and strictly prevents the group from using more. * Soft Limit (
cpu_weight
): Sets a relative priority. If the node is busy, the group gets its fair share; if the node is idle, it can "borrow" more. Example: Creating a group restricted to a portion of CPU:

Copy code

sql
CREATE RESOURCE GROUP restricted_group
PROPERTIES (
    "cpu_weight" = "10", -- Soft priority
    "mem_limit" = "20%",
    "concurrency_limit" = "10"
);

Assign users or queries to this group to ensure they don't spike the whole node. 3. Adjust Query Parallelism (DOP) By default, StarRocks sets

pipeline_dop

to 0, which often uses half of your CPU cores per query fragment. If you have many small queries, this can lead to high CPU context switching. * Global change:

SET GLOBAL pipeline_dop = 8;

(Set this to a lower number to reduce the "burstiness" of individual queries). * Per-session/query:

SELECT /_+ SET_VAR(pipeline_dop = 4) _/ ...

4. Limit CN/BE Internal Thread Pools You can tune the CN's internal configuration (

cn.conf

) to limit how many threads it creates. This provides a "physical" ceiling for CPU usage. Parameter | Recommended Change | Effect

pipeline_exec_thread_pool_thread_num

| Set to 70% of vCPUs | Limits the total threads for the main execution engine.

scanner_thread_pool_thread_num

| Lower from default (usually 48) | Limits threads used for scanning data.

pipeline_connector_scan_thread_num_per_cpu

| Lower from 8 | Limits scan threads specifically for external catalogs (JDBC/HDFS). Summary Recommendation To keep CPU below 70%: 1. Set
query_queue_cpu_used_permille_limit = 700
to stop new queries when the node is busy. 2. Set
pipeline_dop
to a fixed, moderate value (like 4 or 8) instead of letting it scale to all cores. 3. Use Resource Groups to isolate "heavy" users or background tasks (like ETL) from your high-priority dashboard queries. References • administration/management/resource_management/query_queues.md • administration/management/resource_management/resource_group.md • administration/management/BE_configuration.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md

Rocky

04/10/2026, 1:27 AM

• sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md

Philip Oliva

04/10/2026, 1:32 AM

@Rocky I was using v2 Queuing though.

Rocky

04/10/2026, 1:33 AM

If you are using Query Queue V2 (

enable_query_queue_v2 = true

), the mechanism for managing CPU usage is more sophisticated because it uses a slot-based scheduler rather than just a simple concurrency counter. To keep your CN's CPU usage below 70% with V2 Queuing, you should combine the Reactive Threshold (which stops new queries) and the Proactive Slot Tuning (which controls how many queries can run at once). 1. The Reactive Cap (Threshold-based) Even with V2 queuing enabled, the global CPU threshold variable is still the most direct way to "brake" when the node gets hot.

Copy code

sql
-- Trigger queuing when any CN's CPU usage exceeds 70%
SET GLOBAL query_queue_cpu_used_permille_limit = 700;

* How it works in V2: When the FE (Leader) receives a resource report from a CN showing >70% CPU, it will stop assigning new query "slots" to that node and put incoming queries into the V2 pending queue. * Pro Tip: By default, CNs report usage every 1000ms. If your CPU spikes too fast for the queue to react, you can decrease the reporting interval in `cn.conf`:

report_resource_usage_interval_ms = 500

2. The Proactive Cap (Slot Tuning) Query Queue V2 calculates a total number of "slots" available in the cluster. If you want to lower the overall "pressure" on your CPUs, you should reduce the number of available slots. * Reduce Concurrency Level:

Copy code

sql
    -- Default is 4. Reducing this to 2 or 3 will lower the total available slots cluster-wide.
    ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = "2");

Calculation:

Total Slots = concurrency_level _ num_CNs _ CPU_cores_per_CN

. Lowering the level effectively lowers the ceiling of parallel work. * Increase CPU Cost Per Slot:

Copy code

sql
    -- Default is 1,000,000,000.
    -- Increasing this value makes the FE think queries are "heavier,"
    -- causing them to consume more slots and preventing too many from running at once.
    ADMIN SET FRONTEND CONFIG ("query_queue_v2_cpu_costs_per_slot" = "2000000000");

3. Hard CPU Limits (Resource Groups) If you are on version v3.3.5 or higher, V2 queuing integrates perfectly with Resource Group Hard Limits. This is the only way to strictly guarantee a group of queries never exceeds a CPU core count.

Copy code

sql
-- Example: Create a group that is physically capped to 20 cores
-- (If your CN has 32 cores, this effectively keeps that group at ~60% usage)
CREATE RESOURCE GROUP dashboard_users
PROPERTIES (
    "cpu_core_limit" = "20", 
    "mem_limit" = "30%",
    "concurrency_limit" = "20"
);

Summary for your 70% Goal: 1. Immediate fix:

SET GLOBAL query_queue_cpu_used_permille_limit = 700;

2. To prevent "bursting" past 70%: Lower

query_queue_v2_concurrency_level

. 3. For strict isolation: Use a Resource Group with

cpu_core_limit

set to 70% of your CN's total cores. How to Monitor You can check how the V2 queue is behaving with:

Copy code

sql
SHOW PROC '/query_queries'; -- View current pending queries in V2 queue
SHOW USAGE RESOURCE GROUPS; -- See how much CPU each group is actually using on the CNs

References • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md • administration/management/resource_management/resource_group.md References • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md • faq/resource_isolation_faq.md

Rocky

04/10/2026, 1:33 AM

• sql-reference/System_variable.md

Philip Oliva

04/10/2026, 2:04 AM

Rocky

04/10/2026, 2:05 AM

The reason you are seeing spikes up to 90% despite having a 45% limit (

) is primarily due to Reporting Lag and the fact that Query Queues are an Admission Control mechanism, not a CPU limiter. Here is why your current configuration is allowing these spikes: 1. The 5-Second Lag (
query_queue_fresh_resource_usage_interval_ms
) Your output shows:

query_queue_fresh_resource_usage_interval_ms | 5000

This is a very long interval for a high-performance database. It means the FE (which decides whether to queue a query) only refreshes its "view" of the CN's CPU usage every 5 seconds. * The Scenario: If a CN is at 20% CPU, the FE sees "20%" and starts a 5-second timer. * The Spike: During those 5 seconds, you could send 100 heavy queries. The FE will admit all of them because it still thinks the CN is at 20%. * The Result: The CN CPU hits 90%+ instantly, but the FE won't stop admitting queries until the 5-second timer expires and it receives a new report showing the usage is over 45%. Recommendation: Lower this interval significantly to make the queue more reactive.

Copy code

sql
SET GLOBAL query_queue_fresh_resource_usage_interval_ms = 500; -- 500ms

2. Admission vs. Governance The Query Queue only stops new queries from starting. It does not throttle queries that are already running. * If you admit 10 queries that each want 10% of the CPU, your CPU will hit 100%. The queue only prevents the 11th query from starting. * Once a query starts, it will use as much CPU as it can (governed by

pipeline_dop

) to finish quickly. 3. Lack of Concurrency Ceiling You have

query_queue_concurrency_limit | 0

. This means there is no "static" limit. The FE relies entirely on the (delayed) CPU metric. If the CPU metric is stale, there is no second line of defense to stop a flood of queries. Recommendation: Set a realistic concurrency limit as a "hard ceiling" that doesn't depend on metrics.

Copy code

sql
SET GLOBAL query_queue_concurrency_limit = 50; -- Adjust based on your core count

4. How to actually "Cap" the CPU If your goal is to strictly ensure a CN stays below 70%, you cannot rely on the Query Queue alone. You should use Resource Group Hard Limits (available in v3.3.5+). Unlike the queue, a Resource Group with a hard limit physically restricts the number of cores the pipeline engine will allocate to those queries.

Copy code

sql
-- This strictly limits the group to use only 70% of the available cores
CREATE RESOURCE GROUP cap_70_percent
PROPERTIES (
    "cpu_core_limit" = "22", -- Set this to 70% of your CN's total cores
    "mem_limit" = "50%"
);

Summary of recommended changes: 1. Reduce the lag:

SET GLOBAL query_queue_fresh_resource_usage_interval_ms = 500;

2. Sync the CN reporting: In

cn.conf

, ensure

report_resource_usage_interval_ms

is also set to

(default is usually 1000ms). 3. Add a concurrency buffer: Set

query_queue_concurrency_limit

to a non-zero value. 4. Lower Parallelism: If individual queries are too "bursty," lower the global

pipeline_dop

to prevent a single query from grabbing all cores instantly. References • sql-reference/System_variable.md • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md

Philip Oliva

04/10/2026, 2:47 AM

@Rocky setting

query_queue_fresh_resource_usage_interval_ms

to 500 does not seem to stop CPU from reaching 90%.

Rocky

04/10/2026, 2:48 AM

The reason setting

query_queue_fresh_resource_usage_interval_ms

to 500ms isn't stopping the spikes is that Query Queuing is an Admission Control system, not a CPU Governor. It stops new queries from starting, but it cannot slow down a query that has already been admitted. Here is why your CPU is still hitting 90% and how to fix it: 1. The "Blind Spot" (CN Reporting Lag) Even though you told the FE to refresh its resource view every 500ms, the CN (Compute Node) itself only sends its report every 1000ms by default. The FE is effectively checking for new data twice but only seeing a change once every second. * Fix: In your

cn.conf

, add or update:

Copy code

bash
    report_resource_usage_interval_ms = 500

Note: This requires a CN restart to take effect. 2. Admission vs. Execution (The "Burst" Problem) When the CPU is at 40%, the queue allows a new query. If that query is heavy and

pipeline_dop

is set to 0 (default), StarRocks will instantly allocate 50% of the node's cores to that single query. * The CPU jumps from 40% to 90% in milliseconds. * The Queue doesn't see this 90% until the next 500ms report cycle. * By then, the CPU has already been pinned at 90% for half a second. To fix this, lower the "Burstiness" of individual queries:

Copy code

sql
-- Stop queries from being able to grab 50% of the node instantly. 
-- Set this to a fixed small number (e.g., 4 or 8).
SET GLOBAL pipeline_dop = 4;

3. Enable Dynamic Concurrency (Query Queue V2) Your previous output showed

query_queue_driver_high_water | -1

. This means the "V2" part of your queue isn't actually managing the internal parallelism of queries based on load. * Fix: Enable the "Watermarks." This tells StarRocks to automatically lower the

pipeline_dop

of incoming queries as the system gets busier.

Copy code

sql
-- 0 tells the system to use (Total Cores * 16) as the ceiling. 
-- You can also set a specific integer.
SET GLOBAL query_queue_driver_high_water = 0;
SET GLOBAL query_queue_driver_low_water = 0;

4. Use a "Hard Limit" (The only guaranteed way) If you are on v3.3.5 or higher, you can use Exclusive Resource Groups. This is the only feature that physically prevents a group of queries from using more than a set amount of CPU.

Copy code

sql
-- Create a group that physically CANNOT use more than 20 cores 
-- (Assuming a 32-core machine, this is ~60% CPU)
CREATE RESOURCE GROUP capped_group
PROPERTIES (
    "exclusive_cpu_cores" = "20",
    "mem_limit" = "50%"
);

-- Crucial: Disable borrowing so other queries don't "steal" the idle 
-- space and push you back to 100%
UPDATE information_schema.be_configs 
SET VALUE = "false" 
WHERE NAME = "enable_resource_group_cpu_borrowing";

Recommended Strategy for 70%: 1. CN Config: Set

report_resource_usage_interval_ms = 500

cn.conf

. 2. Global Throttling:

SET GLOBAL pipeline_dop = 8;

(or lower). 3. V2 Activation:

SET GLOBAL query_queue_driver_high_water = 0;

. 4. Buffer: Keep your

query_queue_cpu_used_permille_limit

at 450 or 500. This gives the system a "buffer" to handle the queries already in flight before they hit the 70% physical danger zone. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md

Philip Oliva

04/10/2026, 11:41 AM

@Rocky how do you disable cache on CNs?

Rocky

04/10/2026, 11:41 AM

To disable caching on StarRocks Compute Nodes (CNs), you need to address several different cache layers: the Data Cache (used for Data Lake/Shared-data), the Storage Page Cache (used for local storage), and the Query/Result Cache. Depending on whether you want a permanent change or just to bypass cache for a specific session (e.g., for performance testing), use the following methods: 1. Disable Data Cache (Shared-Data / Data Lake) This is the most common "cache" on CNs. It stores data from remote storage (S3/HDFS) on local disks or memory. * Permanent (Node-level): Add these to your

cn.conf

and restart the CN:

Copy code

properties
    # Disables both Page Cache and Block Cache (Data Cache)
    datacache_enable = false
    # Also clear the storage path for the cache
    storage_root_path =

_Note: For versions prior to 3.2, you might also need

starlet_use_star_cache = false

._ * Session-level (No restart required): If you just want to run a "cold" query without clearing the physical cache, run:

Copy code

sql
    -- Bypasses the Data Cache for the current session
    SET enable_scan_datacache = false;
    -- For older versions: SET enable_scan_block_cache = false;

────────── 2. Disable Storage Page Cache This caches recently scanned data pages in memory to accelerate repeated scans of the same data. * Permanent (Node-level): Add to

cn.conf

and restart:

Copy code

properties
    disable_storage_page_cache = true

* Dynamic (No restart required): You can update this live across the cluster without a restart:

Copy code

sql
    UPDATE information_schema.be_configs SET VALUE = "true" WHERE NAME = "disable_storage_page_cache";

* Session-level:

Copy code

sql
    SET skip_page_cache = true;

────────── 3. Disable Query / Result Cache This caches the final or intermediate computation results of a query. * Session-level (It is disabled by default):

Copy code

sql
    SET enable_query_cache = false;

────────── 4. Bypassing Disk Cache (v3.3+) If you want to ensure the query goes directly to the remote storage (S3/HDFS/OSS) and ignores any existing local disk cache:

Copy code

sql
SET skip_local_disk_cache = true;

Summary Checklist for "Cold" Performance Testing: To ensure a query is truly "cold" and hitting the remote storage with no memory assistance, run these before your query:

Copy code

sql
SET enable_query_cache = false;
SET enable_scan_datacache = false;
SET skip_page_cache = true;
SET skip_local_disk_cache = true; -- v3.3+

Pro-Tip: If you want to physically "clear" the cache without a full re-install, you can delete the contents of the

datacache

directory in your CN's storage path and restart the process. References • sql-reference/System_variable.md • using_starrocks/caching/block_cache.md • using_starrocks/caching/query_cache.md • administration/management/BE_configuration.md

Philip Oliva

04/10/2026, 1:55 PM

@Rocky If enable_query_cache is false but datacache_enable is true, what does this mean?

Rocky

04/10/2026, 1:55 PM

This specific combination—`enable_query_cache = false` and `datacache_enable = true`—is actually the standard production configuration for StarRocks shared-data or data lake clusters. Here is exactly what that means for your queries: 1. You are caching "Raw Data," not "Results" * datacache_enable = true
(Data Cache/Block Cache): This caches the raw blocks of data from your remote storage (S3, HDFS, etc.) onto the local disk or memory of the CN. * enable_query_cache = false
(Result Cache): This caches the final output of a SQL query. 2. The Query Execution Flow In this state, when you run the same query twice: 1. Computation is repeated: The CN will re-calculate every join, every aggregation, and every filter. It will not remember the "answer" from last time. 2. Scan is "Hot": When the CN goes to read the data needed for those calculations, it will not go over the network to S3/HDFS. Instead, it will pull the data blocks instantly from its own local SSD or memory. 3. Why this is usually better Most users keep

enable_query_cache

off because it is very "fragile." A Query Cache hit only happens if the SQL is exactly the same and the underlying data hasn't changed at all. By contrast, the Data Cache (

datacache_enable

) is much more powerful because: * Reuse across different queries: If Query A asks for

SUM(sales)

and Query B asks for

AVG(sales)

, the Data Cache helps both because they are both reading the same raw

sales

column. * Resilience: If you add one new row to a table, a Query Cache is typically invalidated (invalidating the whole "result"). However, the Data Cache stays valid for 99.9% of the blocks, only needing to fetch the one new block for the new row. Summary of the "Vibe" With this setup, your CNs are acting as stateful accelerators. They are doing the hard work of calculating the math every time (CPU usage), but they are doing it at "local disk speed" rather than "cloud storage speed" (Low I/O wait). If you are seeing 90% CPU spikes, it's because the computation (joins/aggs) is expensive, even though the data access is fast. The Data Cache has made the I/O so fast that the CPU is now the bottleneck.** References • sql-reference/System_variable.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/query_cache.md • using_starrocks/caching/block_cache.md • data_source/data_cache.md

Philip Oliva

04/10/2026, 1:58 PM

@Rocky So what changes when datacache_enable is true and enable_query_cache is true?

Rocky

04/10/2026, 1:58 PM

When you enable both

datacache_enable

and

enable_query_cache

, you are activating a two-layered defense against latency. They operate at different stages of the query execution and have different impacts on your CPU usage. Here is what happens when both are `true`: 1. The Two Layers of Caching Think of it as a "Finished Goods" vs. "Raw Materials" strategy: * Query Cache (The "Pre-cooked" layer): This caches the intermediate results of aggregations for each tablet. _ _If hit:* The CN doesn't even look at the data files. It just grabs the pre-calculated sums/counts for that tablet. This saves a massive amount of CPU. * Data Cache (The "Local Pantry" layer): This caches the raw blocks of data from S3/HDFS on the CN's local disk/RAM. _ _If hit:* The CN still has to do all the math (the CPU-heavy part), but it pulls the data from local NVMe/RAM instead of the network. This saves I/O wait but keeps CPU high. 2. The Execution Flow (With Both Enabled) When a query arrives, StarRocks follows this "Waterfall" logic: 1. Check Query Cache: Is there a pre-aggregated result for Tablet A? * YES (Hit): Skip the scan. Skip the aggregation math. CPU usage for this tablet is nearly zero. 2. Query Cache Miss? Check Data Cache: Since we don't have the "answer," we must scan the raw data. Is the raw data for Tablet A on our local disk? * YES (Hit): Read the raw data from local SSD at 2GB/s+. The CPU immediately starts crunching the numbers. Because the data arrives so fast, the CPU stays very busy (this is where your 90% spikes often come from). 3. Data Cache Miss? Go to Remote Storage: Read the data from S3/HDFS. The CPU will likely "idle" or wait while the network packets arrive. 4. Final Step: Once the calculation is done, StarRocks saves the "answer" in the Query Cache and the "raw data" in the Data Cache for the next person. 3. Impact on your 90% CPU Issue If you enable both, you will see a "bi-modal" behavior: * Repeat Queries: CPU will drop significantly (to 5-10%) because the Query Cache handles the work. * New/Different Queries on Same Data: CPU will remain at 90%+ because the Data Cache is feeding the CPU raw data faster than it can process it. Summary Comparison Feature | What it Caches | CPU Impact | When it helps Data Cache | Raw data blocks | Increases/Sustains (feeds the fire) | First time a new query touches old data. Query Cache | Pre-aggregated results | Decreases (skips the work) | Second time the exact/similar query runs. Pro-Tip: If your goal is to stop the 90% CPU spikes,

enable_query_cache = true

is your best friend for dashboard users who hit "refresh" frequently. However, for "ad-hoc" users who always change their filters, the Query Cache will often miss, and you must rely on Resource Groups to hard-cap the CPU usage. References • using_starrocks/caching/query_cache.md • sql-reference/System_variable.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md • data_source/data_cache.md

Philip Oliva

04/10/2026, 2:00 PM

@Rocky I don't see what the downside is for

enable_query_cache

being true. Why isn't this production standard?

Rocky

04/10/2026, 2:00 PM

While it might seem like a "free" speed boost, there are very specific engineering trade-offs that prevent

enable_query_cache

from being on by default in every production environment. Here is the "fine print" on why it isn't always the standard: 1. The "First Run" Penalty (Latency Overhead) When the cache is enabled, every "miss" (a query that isn't in the cache yet) has to do extra work. The system must calculate the result AND serialize/write that result into the memory cache. * The Downside: For ad-hoc workloads where queries are rarely repeated, every single query will actually run slightly slower than if the cache were off. You pay a "tax" on every query for a benefit you might never collect. 2. Memory vs. Compute Trade-off The Query Cache lives in the BE/CN memory (defaulting to 512MB, but can be increased). * The Downside: In memory-constrained environments, every MB you give to the Query Cache is a MB taken away from the Scan Buffer or Shuffle Buffer. * If your queries return large result sets (e.g., a

GROUP BY

on a high-cardinality column like

user_id

), the cache will fill up instantly with a few huge results, potentially causing memory pressure or "thrashing" where it constantly deletes old entries to make room for new ones. 3. Limited "Surface Area" (It doesn't work for everything) The Query Cache is highly optimized for Aggregate Queries (SUM, COUNT, etc.). It is much less effective (or completely bypassed) for: * Large Result Sets: If your query returns 500,000 rows of raw data, StarRocks will skip the cache entirely (controlled by

query_cache_entry_max_rows

). * High Parallelism (DOP) vs. Low Tablet Count: If you have more CPU cores (DOP) than tablets in your table, the Query Cache often cannot trigger because the "per-tablet" computation logic is split across too many threads. * Complex Joins: While it has improved (supporting Star Schemas since v3.0), very complex snowflake schemas or non-shuffled joins can still bypass it. 4. Data Freshness & Consistency Complexity StarRocks uses a "Multi-Version" cache, which is very clever—it can even handle append-only data by merging "cached old data" with "newly landed data." * The Downside: For Primary Key or Unique Key tables where data is frequently

UPDATED

DELETED

, the multi-versioning logic is much more restrictive. If your table is constantly churning with updates, the cache hit rate will drop to near zero, making the feature a net-negative for performance. 5. Potential for "Performance Jitter" In production, "stable" performance is often valued over "occasionally blazing fast" performance. * Scenario: A dashboard normally loads in 100ms (Cache Hit). Suddenly, the cache expires or the data changes, and it jumps to 5 seconds (Cache Miss). * This "jitter" can be harder for users to deal with than a consistent 1-second response time. When SHOULD you make it your production standard? You should turn it on if your workload looks like this: 1. High Concurrency: Hundreds of users hitting the same 10-20 dashboards. 2. Duplicate/Similar Queries: Users often look at the same time ranges or the same metrics. 3. Large Tables / Small Results: You are scanning billions of rows but only returning a small aggregate table (like a 10-row summary). 4. Mostly Append-only Data: Your data is logging/telemetry, not frequent updates to old rows. Summary: For the average "Swiss Army Knife" database user, leaving it off ensures predictable memory usage and no "write-tax" on new queries. It's a specialized tool you "opt-in" to when your dashboard traffic starts hitting 90% CPU needs a specific remedy. References • using_starrocks/caching/query_cache.md

Philip Oliva

04/10/2026, 2:12 PM

@Rocky I think tail latency might have got worse when I enabled

enable_query_cache

Rocky

04/10/2026, 2:12 PM

It is very common for tail latency (P99) to increase when

enable_query_cache

is turned on, even if your average (P50) latency gets better. While the "cache hits" are incredibly fast, the "cache misses" become slightly slower than they were before. Since P99 measures your slowest queries, those "slower misses" pull your tail latency upward. Here are the four specific reasons why your P99 is likely suffering: 1. The "First-Run" Tax (Serialization Overhead) When a query misses the cache, StarRocks doesn't just execute it normally. It now has to: * Calculate the intermediate results (normal work). * New work: Serialize those results into a cache-friendly format. * New work: Write those results into the BE’s memory. This extra CPU work and memory copying typically adds 5–15% overhead to every query that isn't already in the cache. If your cache hit rate is low, almost every query is paying this "tax," making your slowest queries even slower. 2. Lock Contention on High Concurrency The Query Cache is a shared structure in the Backend (BE) memory. * When 50 threads all miss the cache at the same time and finish their work simultaneously, they all try to write their results into the LRU cache at once. * This can create "mutex contention" (locking) where threads wait for their turn to update the cache metadata. This waiting time shows up directly in your P99 latency. 3. Frequent Data Invalidation (The "Churn" Problem) The Query Cache is extremely sensitive to data changes. * If you have a table that receives small, frequent trickles of data (upserts or appends), StarRocks has to constantly invalidate old cache entries or attempt to "merge" them with new data versions. * The logic to determine if a cache entry is still valid for a specific data version adds a small amount of "planning time" to the start of every query. In a high-churn environment, you are essentially constantly "warming" the cache but never hitting it. 4. Memory Pressure and GC The Query Cache defaults to 512MB per BE (

query_cache_capacity

). * If your queries return many rows (close to the

query_cache_entry_max_rows

limit), you can fill that 512MB very quickly. * When the cache is full, the system must trigger LRU eviction to make room for new results. If your BE is already under heavy memory pressure, this extra memory management can trigger more aggressive system-level memory reclamation or minor GC pauses, which spike your P99. How to tell if you should keep it off: Check your Query Profile for these metrics: * `QueryCacheHit`: If this is below 50%, the "tax" you are paying on misses is likely outweighing the benefits of the hits. * `QueryCacheWriteTimer`: This tells you exactly how much time was "wasted" writing to the cache on a miss. Recommendation: If you see P99 spikes, try increasing the

query_cache_capacity

be.conf

(e.g., to 2GB or 4GB) to reduce eviction churn, or simply disable it if your workload is mostly "one-and-done" ad-hoc queries where the same result is rarely requested twice. References • using_starrocks/caching/query_cache.md • sql-reference/System_variable.md • administration/management/BE_configuration_list.md References • using_starrocks/caching/query_cache.md • sql-reference/System_variable.md

Philip Oliva

04/10/2026, 2:13 PM

@Rocky How do I increase memory that the cache is allowed to use on CNs?

Rocky

04/10/2026, 2:14 PM

To increase the memory for these two caches on your Compute Nodes (CNs), you need to modify two different sets of parameters, as they manage memory differently. 1. Increasing Query Cache Memory The Query Cache stores pre-aggregated results in the CN's RAM. * Parameter:

query_cache_capacity

* Default:

512MB

* How to change: 1. Open your

cn.conf

(or

be.conf

if deployed as BE). 2. Add or modify the line:

query_cache_capacity = 2147483648

(to set it to 2GB, value is in bytes). 3. Restart the CN process. This parameter cannot be changed dynamically. * Best Practice: Do not exceed 1/6 of the total memory provisioned to the CN. ────────── 2. Increasing Data Cache Memory The Data Cache uses memory primarily for metadata and as a "buffer" for raw data blocks being pulled from remote storage (S3/HDFS) to the local disk. * Parameter:

datacache_mem_size

* How to change: You can do this dynamically without a restart using SQL, or permanently in the config file. * Option A: Dynamic (Immediate but lost on restart)

Copy code

sql
        -- Set to a specific size (e.g., 10GB)
        UPDATE be_configs SET VALUE="10G" WHERE NAME="datacache_mem_size";

        -- Or set as a percentage of total CN memory
        UPDATE be_configs SET VALUE="10%" WHERE NAME="datacache_mem_size";

* Option B: Permanent (Requires restart) Add

datacache_mem_size = 10G

to your

cn.conf

query_cache_capacity

| No | 512MB | < 1/6 of RAM Data Cache |

datacache_mem_size

| Yes | ~5% of RAM | 5% - 10% of RAM Important Tip for CNs Since CNs are often "stateless," ensure that your

storage_root_path

cn.conf

points to a high-performance local NVMe SSD. If you increase the memory but the underlying disk cache is slow, the memory usage will climb as it waits for I/O, leading to the "High memory usage" known issue mentioned in official docs. To verify your changes: Run the following SQL to see the current effective settings:

Copy code

sql
ADMIN SHOW FRONTEND CONFIG LIKE '%query_cache%'; -- For FE-side variables
SHOW BACKEND CONFIG LIKE '%cache%';              -- For CN/BE-side limits

References • using_starrocks/caching/query_cache.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration_list.md References • using_starrocks/caching/query_cache.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md

Philip Oliva

04/10/2026, 2:15 PM

@Rocky what is vector_query_cache_capacity?

Rocky

04/10/2026, 2:15 PM

It appears you may be combining two different StarRocks concepts: the Query Cache and the Vectorized Engine (or the new Vector Search feature). There is no configuration parameter explicitly named

vector_query_cache_capacity

in StarRocks. Depending on what you are trying to tune, you are likely looking for one of the following three parameters: 1.
query_cache_capacity
(Most Likely) This is the setting that controls the memory limit for the Query Cache (which stores the results of aggregated queries). * Location:

be.conf

cn.conf

. * Default:

512MB

(536870912 bytes). * Purpose: Since StarRocks uses a vectorized pipeline engine, many users colloquially refer to this as the "vector query cache," but its official name is simply the Query Cache. * Note: If you increase this, remember it must be in bytes (e.g.,

2147483648

for 2GB) and requires a restart. 2.
vector_chunk_size
If you are looking for a parameter starting with "vector" that affects memory and performance, this is often it. * Default:

. * Purpose: It defines the number of rows in a "chunk" processed by the vectorized engine. * Relation to Cache: Larger chunk sizes can improve CPU efficiency but increase the memory footprint of every operator in the query plan. 3. Vector Search (ANN) Memory Management (v3.4+) If you are using the new Vector Index (for similarity search with

ARRAY<FLOAT>

columns), memory is handled differently: * IVFPQ Indexes: These allow for a "caching strategy" where you can balance memory vs. latency by adjusting how much of the index block is cached. * HNSW Indexes: These currently require full-file caching in memory to perform efficiently. * Configuration: These don't have a single "capacity" toggle; they primarily consume memory from the Page Cache (controlled by

datacache_mem_size

or the older

storage_page_cache_limit

). Which one should you change? * If you want to store more "Cache Hits" for your dashboards: Increase

query_cache_capacity

. * If you are doing Similarity Search (AI/LLM) and it's slow: Ensure

enable_experimental_vector

is true and check your Page Cache (

datacache_mem_size

) to ensure the vector index files aren't being evicted. * If you see "Delete Vector" memory growing: This refers to Primary Key table metadata. You can monitor this with the metric

update_del_vector_bytes_total

, but it is managed automatically by the system's memory tracker. Summary: If you saw the term

vector_query_cache_capacity

in a blog post or older discussion, it was likely a typo for query_cache_capacity
. References • using_starrocks/caching/query_cache.md • administration/management/BE_configuration.md • table_design/indexes/vector_index.md References • table_design/indexes/vector_index.md

Philip Oliva

04/10/2026, 2:16 PM

mysql> SELECT BE_ID, NAME, VALUE,

DEFAULT

, MUTABLE, TYPE FROM information_schema.be_configs WHERE NAME LIKE '%query_cache_capacity%'; +--------+-----------------------------+-----------+-----------+---------+-------+ | BE_ID | NAME | VALUE | DEFAULT | MUTABLE | TYPE | +--------+-----------------------------+-----------+-----------+---------+-------+ | 107640 | query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107640 | vector_query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107581 | query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107581 | vector_query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107598 | query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107598 | vector_query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107619 | query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107619 | vector_query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107562 | query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107562 | vector_query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107448 | query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107448 | vector_query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107447 | query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 107447 | vector_query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 10001 | query_cache_capacity | 536870912 | 536870912 | 0 | int64 | | 10001 | vector_query_cache_capacity | 536870912 | 536870912 | 0 | int64 | +--------+-----------------------------+-----------+-----------+---------+-------+ 16 rows in set (0.129 sec)

Philip Oliva

04/10/2026, 2:16 PM

@Rocky mysql> SELECT BE_ID, NAME, VALUE,

DEFAULT

Rocky

04/10/2026, 2:16 PM

Based on the output you shared, you are seeing both query_cache_capacity
and vector_query_cache_capacity
because they are essentially aliases for the same underlying configuration in newer versions of StarRocks. What is the difference? There is no functional difference between the two in modern StarRocks (v3.x+). * query_cache_capacity
: The original parameter name used when the feature was first introduced. * vector_query_cache_capacity
: A more descriptive internal name introduced to clarify that this memory pool belongs to the Vectorized Pipeline Engine's intermediate result cache (the "Multilane" Query Cache). Because StarRocks strives for backward compatibility, both names are exposed in the

be_configs

table. If you modify one in your

be.conf

, you will typically see both reflect the same value upon restart, as they point to the same memory manager in the Backend (BE) or Compute Node (CN). Why do they exist together? 1. Refactoring: As StarRocks moved fully toward the Pipeline Engine, internal names were updated to reflect that the cache stores "Vectorized Chunks" rather than traditional row-based results. 2. Compatibility: If a user had

query_cache_capacity

in their old config files, the system still needs to respect it without forcing a config rewrite during an upgrade. Key Points for Tuning: * Value: Both default to

512MB

(

536870912

bytes). * Mutable: Note the

MUTABLE = 0

in your output. This means you cannot change these via SQL (

SET GLOBAL

UPDATE be_configs

). You must edit the

be.conf

cn.conf

file and restart the node. * Recommendation: When updating your config, it is safer to use the standard name

query_cache_capacity

, as that is what is documented in the official manuals. Which "Cache" are you actually using? To be sure which one is affecting your tail latency, check your Query Profile: * If you see CacheOperator
metrics, you are using the cache controlled by these two capacity parameters. * If you see ResultCache
(without the "Vector" or "Multilane" prefix), that is the FE-side Result Cache, which is separate and generally has much lower overhead. Summary: You don't need to tune them separately. Increasing

query_cache_capacity

in your config will increase the memory available for both entries you see in that table. References • using_starrocks/caching/query_cache.md • administration/management/BE_configuration_list.md References • using_starrocks/caching/query_cache.md • administration/management/BE_configuration.md • sql-reference/System_variable.md • administration/management/monitoring/metrics.md

Philip Oliva

04/10/2026, 2:19 PM

@Rocky

datacache_mem_size

has value 0 currently... what does that mean?

Rocky

04/10/2026, 2:20 PM

In StarRocks, when datacache_mem_size
is set to 0
, it typically means the system is using an automatic default or deferring to legacy parameters, depending on your version. Here is the breakdown of what that

means and how it affects your CN: 1. It Defaults to a Percentage (v3.2.x and later) In most recent versions, a value of

does not mean "zero memory." Instead, the system often defaults to an internal calculation—typically 10% of the total memory provisioned to the node. * If you haven't explicitly set a value, StarRocks allows the Data Cache (which handles both local tables and Data Lake access) to use a small slice of RAM to speed up metadata lookups and small data blocks. 2. Legacy Parameter Interaction (
storage_page_cache_limit
) Before the modern "Data Cache" framework was unified, StarRocks used

storage_page_cache_limit

to control memory for caching. * If

datacache_mem_size

, the system may still be respecting the older storage_page_cache_limit
, which defaults to 20% of total memory. * Rocky's Tip: In newer versions (v3.3+ and v4.0),

storage_page_cache_limit

is being deprecated in favor of

datacache_mem_size

. 3. Shared-Data vs. Shared-Nothing * On Compute Nodes (CNs): CNs are designed for "Shared-Data" (Data Lake) architecture. Here, the

datacache_mem_size

is critical because it caches data from remote storage (like S3/HDFS). If it's

, you are likely relying entirely on the Disk Cache (controlled by

datacache_disk_size

) or a very small default RAM buffer. * Impact: If your

datacache_mem_size

, you might see higher "tail latency" for queries because the system has to fetch more "metadata" or small file footers from the disk cache or remote storage instead of RAM. How to change it Since you saw

MUTABLE = 1

(or "Yes") in the

be_configs

table for this parameter, you can change it dynamically without a restart to see if it improves performance:

Copy code

sql
-- Set to a specific percentage of node RAM
UPDATE be_configs SET VALUE = "20%" WHERE NAME = "datacache_mem_size";

-- OR set to a specific physical size
UPDATE be_configs SET VALUE = "16G" WHERE NAME = "datacache_mem_size";

Recommendation If you have a high-memory CN and are querying large datasets from S3/HDFS/Iceberg: 1. Don't leave it at 0. Explicitly set it to 10% - 20% of your RAM. 2. Monitor the Hit Rate: After changing it, check your metrics for

page_cache_hit_rate

block_cache_hit_rate

. If the hit rate remains low despite increasing memory, your bottleneck might be the disk cache (

datacache_disk_size

) or the network, not the RAM cache. References • data_source/data_cache.md • administration/management/BE_configuration_list.md • using_starrocks/caching/block_cache.md

Philip Oliva

04/10/2026, 2:29 PM

@Rocky why on DataCacheMetrics does MemUsage: 0B/18.1GB have 18.1 GB Max? Is that a setting? *************************** 2. row *************************** ComputeNodeId: 107447 IP: star-rocks-cluster-2-cn-1.star-rocks-cluster-2-cn-search.star-rocks-cluster-2.svc.cluster.local HeartbeatPort: 9050 BePort: 9060 HttpPort: 8040 BrpcPort: 8060 LastStartTime: 2026-04-10 102731 LastHeartbeat: 2026-04-10 102741 Alive: true SystemDecommissioned: false ClusterDecommissioned: false ErrMsg: Version: 3.5.15-5abb1cb CpuCores: 62 MemLimit: 90.720GB NumRunningQueries: 0 MemUsedPct: 0.28 % CpuUsedPct: 0.0 % DataCacheMetrics: Status: Normal, DiskUsage: 16GB/810GB, MemUsage: 0B/18.1GB HasStoragePath: true StatusCode: OK StarletPort: 9070 WorkerId: 25011 WarehouseName: default_warehouse TabletNum: 229

Rocky

04/10/2026, 2:29 PM

The 18.1 GB maximum you see in your

DataCacheMetrics

is not a random number—it is exactly 20% of your total
MemLimit
(90.720 GB). Here is why it shows up that way and what it means for your configuration: 1. The 20% Default Rule Even if you see

datacache_mem_size

set to

, StarRocks does not disable the cache. Instead, it falls back to an internal default calculation. * The Math: $90.720 \text{ GB} \times 0.20 = 18.144 \text{ GB}$. * The Setting: Historically, this was controlled by a parameter called

storage_page_cache_limit

. In modern versions (like your v3.5.15), if

datacache_mem_size

is not explicitly defined (set to 0), the system automatically allocates 20% of the BE/CN process memory limit to the Data Cache. 2. Why is MemUsage "0B"?

MemUsage: 0B/18.1GB

means that while the system has reserved a quota of 18.1 GB, it is currently using 0 bytes of it. This typically happens because: * The node was recently restarted: The cache is "cold" and hasn't loaded any data from the Data Lake or local storage yet. * No "Cacheable" queries have run: The Data Cache primarily caches data from external tables (Iceberg, S3, HDFS) or "Shared-Data" architecture tables. If you are only querying local Olap tables without specific settings, you might not see this fill up immediately. * Page Cache vs. Block Cache: The "MemUsage" here specifically refers to the RAM portion (Page Cache). If your queries are hitting the Disk portion (Block Cache), you will see the

DiskUsage

increase (which is currently at 16GB/810GB in your output) while RAM usage stays low until specific data blocks are promoted to memory. 3. Is this a setting you can change? Yes. If you want to change that 18.1 GB limit to something else, you have two options: Option A: Set a specific size (e.g., 32GB)

Copy code

sql
-- This overrides the 20% default with a fixed value
UPDATE be_configs SET VALUE = "32G" WHERE NAME = "datacache_mem_size";

Option B: Set a specific percentage (e.g., 10%)

Copy code

sql
-- This will change the max to ~9GB in your case
UPDATE be_configs SET VALUE = "10%" WHERE NAME = "datacache_mem_size";

Summary The 18.1 GB is the automatic safety limit StarRocks calculated to ensure your Data Cache has room to grow without starving the query execution engine of memory. It represents the "ceiling" of how much RAM your Data Cache is allowed to consume. References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • release_notes/release-2.4.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md

Philip Oliva

04/10/2026, 2:48 PM

@Rocky why do I always see MemUsage showing 0B?

Rocky

04/10/2026, 2:49 PM

The reason you see MemUsage: 0B
even though your

DiskUsage

is populated (16GB) and you have a 18.1GB "Max" limit is that the memory-tier of the Data Cache is currently disabled by your configuration. Here is the technical breakdown: 1.
datacache_mem_size = 0
Disables the RAM Tier In StarRocks 3.x, the Data Cache (also called the Block Cache) has two tiers: Memory (RAM) and Disk. * The Disk Tier: This is enabled by default if you have a

storage_root_path

configured or explicit

datacache_disk_size

settings. This is why you see

16GB/810GB

being used—your queries are successfully caching data from S3/HDFS/Iceberg onto your local NVMe/SSD disks. * The Memory Tier: This is controlled by datacache_mem_size
, which defaults to 0
. When this is set to 0, the system tells the cache: "Do not store any data blocks in RAM; send everything directly to the disk cache." 2. Why does it show
/ 18.1GB
then? The

18.1GB

is the Quota Ceiling. Even though you haven't enabled the RAM cache yet, StarRocks calculates a "safety limit" (which you correctly identified as 20% of your 90GB MemLimit). _ This limit is reserved by the system to ensure that if you _did* enable the cache, it wouldn't crash the node by taking too much memory. _ It effectively says: _"The capacity is theoretically 18.1GB based on system defaults, but the active setting is 0B."* 3. Page Cache vs. Data Cache There is often confusion between the two: * Page Cache: Used for internal OLAP tables (Shared-Nothing). This uses RAM automatically. * Data Cache (Block Cache): Used for Data Lake / Shared-Data tables. This is what the

DataCacheMetrics

string specifically monitors. If you are querying a Data Lake (Iceberg/Hudi/S3), the data will only show up in

MemUsage

datacache_mem_size

is greater than 0. How to fix it If you want to speed up your queries by using that 18GB of RAM as a "hot" layer for your Data Lake data, you need to change the setting:

Copy code

sql
-- Change it to 10GB or 10% to start seeing MemUsage grow
UPDATE be_configs SET VALUE = "10G" WHERE NAME = "datacache_mem_size";

Once you run this: 1. The next time you query your external tables, StarRocks will fill the RAM tier first (or simultaneously). 2. The

MemUsage

DataCacheMetrics

will start showing values like

2.4GB / 18.1GB

. 3. Your "First Read" latency for hot data will drop significantly because it will come from RAM instead of your local disk. References • data_source/data_cache.md • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • data_source/data_cache_observe.md References • administration/management/BE_configuration.md • data_source/data_cache_observe.md

Philip Oliva

04/10/2026, 2:51 PM

@Rocky I already have that at 20% so why is nothing going to cache memory? mysql> SELECT BE_ID, NAME, VALUE,

DEFAULT

, MUTABLE, TYPE FROM information_schema.be_configs WHERE NAME LIKE '%datacache_mem_size%'; +--------+--------------------+-------+---------+---------+--------+ | BE_ID | NAME | VALUE | DEFAULT | MUTABLE | TYPE | +--------+--------------------+-------+---------+---------+--------+ | 10001 | datacache_mem_size | 20% | 0 | 1 | string | | 107447 | datacache_mem_size | 20% | 0 | 1 | string | | 107448 | datacache_mem_size | 20% | 0 | 1 | string | +--------+--------------------+-------+---------+---------+--------+ 3 rows in set (0.058 sec)

Rocky

04/10/2026, 2:51 PM

Even though you have set

datacache_mem_size

20%

(calculating to 18.1 GB), you see MemUsage: 0B
because of how StarRocks v3.x splits its caching duties between RAM and Disk. In your version (v3.5.15), the Data Cache is actually composed of two distinct layers that handle different types of data: 1. The "DiskUsage" is your External Data (Block Cache) The

DiskUsage: 16GB/810GB

you see is the Block Cache. * This is where StarRocks stores the raw data blocks it pulls from S3, HDFS, or Iceberg. * Default Behavior: For external tables (Data Lake), StarRocks caches data to Disk by default. It does not automatically "double-cache" these blocks into RAM because doing so would consume massive amounts of memory that are usually better spent on query execution (shuffles, joins). 2. The "MemUsage" is the Page Cache The

MemUsage: 0B / 18.1GB

tracks the Page Cache. This memory is reserved for three specific things: * Internal Tables: If you have local StarRocks tables (OLAP engine), their decompressed data pages are stored here. * File Footers: Metadata about your external files (Parquet/ORC footers) is stored here to avoid re-reading file headers. * Decompressed Pages: In some specific configurations, StarRocks can store decompressed pages of external data here, but this is usually disabled or rarely triggered for standard Data Lake queries. Why is it still 0B? If you are primarily querying a Data Lake (External Catalog) and not native StarRocks tables: 1. No Internal Tables: You aren't querying any local

DUPLICATE

PRIMARY KEY

tables that would populate the RAM cache. 2. Metadata is Small: File footers (metadata) are extremely small. Even if you have thousands of files, they might only take up a few Megabytes, which often rounds to

0B

0.00GB

in the summary metrics. 3. Default Session Settings: The session variable

use_page_cache

is often set to

false

by default for external table data to prevent one big scan from evicting all other useful memory. How to verify it is working To see if anything is going into that memory, try querying a local StarRocks table or run a query on an external table and check your BE metrics: 1. Check the Page Cache Hit Rate:

Copy code

sql
    -- Check if Page Cache is being touched at all
    SELECT * FROM information_schema.be_datacache_metrics;

2. Force Memory Caching (Not Recommended for all data): If you really want to see that number move, you can try enabling the page cache for your session, though StarRocks usually manages this automatically for what it deems "useful":

Copy code

sql
    SET use_page_cache = true;

Summary Your configuration is correct. The

18.1GB

is the "ceiling" (quota) you've allowed the RAM cache to reach. The

0B

just means that for your current workload (likely external Data Lake queries), StarRocks is efficiently using the Disk Cache (16GB used) and hasn't found a reason to move large amounts of data into the RAM tier yet. Rocky's Tip: In StarRocks v4.0, these two tiers are being unified even further to make this distinction less confusing! References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • data_source/data_cache.md • knowledge/usage/best-practice.md • sql-reference/System_variable.md • using_starrocks/caching/query_cache.md

Philip Oliva

04/10/2026, 3:06 PM

@Rocky use_page_cache is already set to true. Is it maybe because everything is already in disk cache? Do I need to wipe cache on disk and rerun tests? mysql> SHOW GLOBAL VARIABLES LIKE "%use_page_cache%"; +----------------+-------+ | Variable_name | Value | +----------------+-------+ | use_page_cache | true | +----------------+-------+ 1 row in set (0.039 sec)

Rocky

04/10/2026, 3:07 PM

The reason you see MemUsage: 0B / 18.1GB
while DiskUsage: 16GB / 810GB
is already populated comes down to how StarRocks (v3.x) manages its two-tier caching system for Data Lake/External queries. 1. The Two Layers of Data Cache In your version (v3.5.15), the "Data Cache" is actually divided into two distinct components: * Block Cache (Disk Layer): This is where StarRocks stores the 1MB data blocks it pulls from your Data Lake (S3, HDFS, Iceberg, etc.). This is shown as

DiskUsage

. Your 16GB usage proves that your cache is working and correctly populating your local disks. * Page Cache (Memory Layer): This is what

MemUsage

tracks. It is specifically designed to store decompressed data pages and file footers (metadata). 2. Why is MemUsage "0B"? Even though you have

use_page_cache = true

and a

20%

(18.1GB) quota, you see 0B because: * External Data Priority: For external tables (Data Lake), StarRocks prioritizes the Disk-based Block Cache. It generally avoids caching the actual data pages of external tables in RAM to prevent query execution from running out of memory (OOM). * Footers are Tiny: The only thing StarRocks usually puts in the memory tier for external tables are the Parquet/ORC File Footers. Even with hundreds of files, these are so small (often just a few MBs) that they round down to

0B

in the summary metrics. * Internal Tables: The memory tier is primarily used for Internal OLAP tables (Shared-Nothing). If you aren't querying local

DUPLICATE

PRIMARY KEY

tables, the memory tier remains mostly empty. 3. Do you need to wipe the disk cache? No. Wiping the disk cache will not help. If you wipe the disk, StarRocks will simply fetch the data from the remote source and put it back on the disk. It will not "promote" it to RAM unless it's an internal table or specifically required metadata. 4. How to verify it's actually "on"? To see if that 18.1GB is actually functional, you can check the detailed metrics in the `information_schema`:

Copy code

sql
SELECT * FROM information_schema.be_datacache_metrics;

Look for

page_cache_mem_used_bytes

. If it shows a small non-zero value, the memory tier is working for metadata. Summary * The 18.1 GB Max: Is indeed your

20%

setting. It's the "ceiling" you've allowed the RAM cache to reach. * The 0B Usage: Is normal for Data Lake workloads. Your speed gains are coming from the 16GB on Disk, which is significantly faster than fetching from S3/HDFS. If you want to use RAM specifically to speed up external table queries, you would typically look at Query Cache (

enable_query_cache = true

) rather than the Data Cache's memory tier, as the Query Cache stores the final result sets in memory. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • data_source/data_cache_observe.md • administration/management/BE_configuration.md • using_starrocks/caching/query_cache.md • data_source/data_cache_troubleshooting.md References • using_starrocks/caching/block_cache.md • data_source/data_cache.md

Philip Oliva

04/10/2026, 3:09 PM

@Rocky mysql> SELECT * FROM information_schema.be_datacache_metrics; +--------+--------+------------------+-----------------+-----------------+----------------+-----------------+-----------------------------------------------------------------------------+---------------------+ | BE_ID | STATUS | DISK_QUOTA_BYTES | DISK_USED_BYTES | MEM_QUOTA_BYTES | MEM_USED_BYTES | META_USED_BYTES | DIR_SPACES | USED_BYTES_DETAIL | +--------+--------+------------------+-----------------+-----------------+----------------+-----------------+-----------------------------------------------------------------------------+---------------------+ | 107448 | Normal | 869730877440 | 17178820608 | 19481971654 | 0 | 15711312 | [{"path":"/opt/starrocks/cn/storage/datacache","quota_bytes":869730877440}] | {014717311941,10} | | 107447 | Normal | 869730877440 | 17243832320 | 19481971654 | 0 | 16064063 | [{"path":"/opt/starrocks/cn/storage/datacache","quota_bytes":869730877440}] | {014630174025,10} | | 10001 | Normal | 869730877440 | 17054302208 | 19481971654 | 0 | 15131242 | [{"path":"/opt/starrocks/cn/storage/datacache","quota_bytes":869730877440}] | {014671548913,10} | +--------+--------+------------------+-----------------+-----------------+----------------+-----------------+-----------------------------------------------------------------------------+---------------------+ 3 rows in set (0.212 sec)

Rocky

04/10/2026, 3:10 PM

The behavior you are seeing—*`DiskUsage`* populating (17GB) while MemUsage
remains 0B
(despite a 18.1GB quota)—is expected in StarRocks 3.x when querying External Tables (Data Lake). The technical reason lies in how StarRocks separates its caching layers: 1. Block Cache (Disk) vs. Page Cache (RAM) In StarRocks v3.x, the "Data Cache" system is actually composed of two distinct components that handle different types of data: * Block Cache (
DiskUsage
): This is the primary cache for External Tables (S3, HDFS, Iceberg, etc.). It stores the raw data blocks (usually 1MB chunks) exactly as they are pulled from remote storage. This cache is disk-based. This is where your 17GB is currently living. * Page Cache (
MemUsage
): This is the memory tier controlled by

datacache_mem_size

. It is designed to store decompressed data pages. * Native Tables: It is heavily used for StarRocks' own internal OLAP tables. * External Tables: For Data Lake queries, this tier is primarily used for Metadata (File Footers). 2. Why is
MemUsage
showing
0B
? Even with

use_page_cache = true

, you see

0B

because: 1. Metadata is small: Your

be_datacache_metrics

output shows

META_USED_BYTES

at 15MB. In the high-level summary string (e.g., in

SHOW BACKENDS

), 15MB is often rounded down or displayed as

0B

0.01GB

because it is negligible compared to your 18.1GB quota. 2. External Data stays on Disk: By default, StarRocks does not cache the actual "Data" of external tables in RAM. It caches the raw blocks on Disk (Block Cache) to save memory for query execution (joins, shuffles). It only puts file metadata (footers) in the RAM tier. 3. Decompression overhead: Page Cache only stores decompressed pages. For many Data Lake workloads, StarRocks determines that the overhead of maintaining a RAM cache for transient data pages isn't worth the memory pressure, preferring the high-speed NVMe/SSD "Block Cache" instead. 3. Your
USED_BYTES_DETAIL
Analysis In your output: `USED_BYTES_DETAIL: {0:14671548913, 1:0}`: * Index
0
is the Disk Tier (Block Cache). It is holding your ~14.6GB of data. * Index
1
is the Memory Tier (Page Cache). It is currently holding 0 bytes of data pages. Does this mean it's not working? No, it is working perfectly. Your queries are hitting the 17GB Disk Cache, which is significantly faster than fetching from S3/HDFS. How to make the memory tier work? If you want to see that

MemUsage

number grow, you can try: 1. Querying Internal Tables: Create a local StarRocks table and query it; you will see the

MemUsage

rise immediately. 2. Wait for high-frequency hits: In some versions, StarRocks may promote frequently accessed external metadata or specific footers to the memory tier, but it will never match the volume of the disk cache. 3. Check Query Cache: If your goal is "Instant RAM performance" for the same query, enable the Query Cache (which stores the final result set in RAM):

Copy code

sql
    SET GLOBAL enable_query_cache = true;

Summary: For Data Lake/External Catalog users, it is perfectly normal for

DiskUsage

to be large and

MemUsage

to stay at or near

0B

. Your performance gains are coming from the local disk cache. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md

Philip Oliva

04/10/2026, 3:47 PM

@Rocky "ProgrammingError(1064, "Query reached its timeout of 300 seconds, please increase the 'query_timeout' session variable, pending time:4")" What does pending time:4 mean?

Rocky

04/10/2026, 3:47 PM

In the StarRocks error message, pending time:4
(usually in seconds) indicates the amount of time the query spent waiting in a Query Queue before it actually began execution. Here is a breakdown of what that means for your query: 1. The Meaning of "Pending" When your StarRocks cluster reaches certain resource limits (CPU usage, Memory usage, or maximum Concurrency), it doesn't always reject new queries immediately. Instead, if Query Queues are enabled, it places new queries in a "PENDING" state. * Total Time (300s): Your query was alive for 300 seconds total. * Pending Time (4s): Out of those 300 seconds, the query sat in a queue for 4 seconds waiting for resources to become available. * Execution Time (296s): The query actually ran on the backends for the remaining 296 seconds before hitting the

query_timeout

limit. 2. Why did it wait? A query enters the pending state if your cluster has hit one of the following thresholds (configured in your Resource Group or global variables): * query_queue_concurrency_limit
: Too many queries are running at once. * query_queue_mem_used_pct_limit
: The cluster is low on memory (default is often 90%). * query_queue_cpu_used_permille_limit
: The CPU is too busy. 3. How to check queue status You can see if other queries are currently waiting by running:

Copy code

sql
-- Check the 'IsPending' column
SHOW PROCESSLIST;

-- Or see specific queue metrics (v3.1.4+)
SHOW RUNNING QUERIES;

4. How to fix it Since your query spent 296 seconds actually executing (300 total - 4 pending), the bottleneck isn't the queue wait—it's the query execution itself. * Immediate Fix: Increase the session timeout for that specific query:

Copy code

sql
    SET query_timeout = 600; -- Increase to 10 minutes
    SELECT ...

* Long-term Fix: If you see "pending time" increasing (e.g., pending time: 50), it means your cluster is overloaded. You should then investigate your Resource Group settings or check if a "big query" is hogging all the CPU/Memory. Rocky's Tip: If the

pending time

ever equals your total

query_timeout

, it means the query timed out without ever even starting! starting. In your case, it did run, but it simply wasn't fast enough to finish in 5 minutes. References • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • administration/management/monitor_manage_big_queries.md References • sql-reference/System_variable.md • administration/management/resource_management/query_queues.md • faq/operation_maintenance_faq.md • faq/Others.md • administration/management/monitor_manage_big_queries.md

Philip Oliva

04/10/2026, 4:25 PM

@Rocky why would query latency appear to not really improve with CN cache enabled?

Rocky

04/10/2026, 4:26 PM

If you have enabled the Data Cache (CN cache) but aren't seeing the expected performance boost, it is usually due to one of the following technical reasons. You can verify each of these using the Query Profile. 1. The "Asynchronous Population" Rule By default, StarRocks populates the cache asynchronously to avoid slowing down the very first query that touches the data. * Behavior: The first time you run a query, StarRocks fetches data from S3/HDFS and subsequently starts a background task to write it to your local disk. * Solution: You often need to run the same query 2 or 3 times before the data is fully "warm" in the cache. * Check: In the query profile, look at

DataCacheWriteBytes

. If it’s high, it means the query is currently populating the cache. If

DataCacheReadBytes

is high, it means it’s finally hitting the cache. 2. Population Rules (v3.3.2+) Starting in newer versions, StarRocks introduced rules to prevent the cache from being "polluted" by one-off giant scans. The cache will not be populated if: _ The query is a

SELECT _

(scans all columns). * The query scans all partitions of a multi-partition table. * Verify: Run

EXPLAIN VERBOSE <your_query>;

. Look for the

dataCacheOptions

section in the scan node. If you see

{populate: false}

, the cache is being bypassed. * Fix: Set

SET populate_datacache_mode = 'always';

to force population regardless of the query pattern. 3. Disk Performance & I/O Adaptor The Data Cache (Block Cache) is stored on the local disks of your CN nodes. * Disk Type: If your CN nodes are using slow HDDs or standard cloud network storage (instead of local NVMe SSDs), the latency of reading from the "cache" might be similar to reading from S3. * I/O Adaptor: StarRocks has an "I/O Adaptor" enabled by default (

enable_datacache_io_adaptor

). If the local disk is busy or has high latency, the system will bypass the cache and fetch from S3 directly to maintain query speed. * Check: Look for

fslib read io_latency

in your BE metrics or check if

DataCacheReadBytes

is significantly lower than the total

BytesRead

. 4. Routing Stability (Consistent Hashing) For the cache to work, the same data must be processed by the same CN node every time. * StarRocks uses Consistent Hashing to assign "scan ranges" to specific nodes. * If your CN cluster is scaling up/down or if nodes are frequently disconnecting, the "ownership" of data blocks shifts to different nodes. This causes a "cache miss" because the new node doesn't have the data that the old node cached. * Solution: Enable Cache Sharing so nodes can pull cache from each other via the network:

Copy code

sql
    SET GLOBAL enable_datacache_sharing = true;

5. JNI vs. Native Reader Data Cache only works for Native File Readers (Parquet, ORC, CSV). * If you are using a JDBC Catalog or certain versions of the Paimon Catalog that fallback to JNI (Java Native Interface) to read data, the Data Cache will not be used. Summary Checklist 1. Check the Profile: Does

DataCacheReadBytes

roughly equal

BytesRead

? If not, you are missing the cache. 2. Run 3 Times: Does the latency drop on the 3rd run? 3. Check Explain: Does

EXPLAIN VERBOSE

show

populate: true

? 4. Check Variables:

Copy code

sql
    SET enable_scan_datacache = true;
    SET populate_datacache_mode = 'always';

DataCacheReadBytes

is high but latency is still high, the bottleneck is likely CPU or Shuffle (network transfer between nodes) rather than the data-reading step. References • table_design/data_distribution/Data_distribution.md • using_starrocks/Colocate_join.md • administration/management/resource_management/Replica.md

Rocky

04/10/2026, 4:26 PM

• introduction/Architecture.md

Philip Oliva

04/10/2026, 4:37 PM

@Rocky Is there way to see in logs if enable_datacache_io_adaptor occured or something we can track on disks?

Rocky

04/10/2026, 4:37 PM

To see if the enable_datacache_io_adaptor
logic has been triggered, you can monitor specific BE metrics and look for tell-tale patterns in the Query Profile or logs. 1. Prometheus Metrics (The Most Reliable Way) StarRocks exposes metrics on the BE/CN

/metrics

endpoint (port 8040 by default). If the I/O adaptor is bypassing the local cache disk due to high latency, you will see it in these counters: * starrocks_be_block_cache_read_local_latency
: This is the metric the adaptor monitors. If this latency (usually measured in microseconds) exceeds a internal threshold, the adaptor starts "skipping" the local disk to fetch from the remote storage instead. * starrocks_be_block_cache_read_remote_bytes
: If you see this incrementing for data that you know is already in the cache, it’s a strong sign the adaptor is routing requests to the network to maintain query speed. * fslib_read_io_latency
: This is the high-level metric for Data Cache disk latency mentioned in the official docs. 2. Query Profile Analysis When you run a query with

SET enable_profile = true;

, look at the Scan Node metrics in the profile. Compare these two: * DataCacheReadBytes
: Data successfully read from your local SSD/NVMe. * BytesRead
: The total data read. * The "Gap": If a query is "warm" (run 3+ times) and

DataCacheReadBytes

is significantly lower than

BytesRead

, and you see a high RemoteReadTimer
, it means the I/O Adaptor likely stepped in because the local disk was too slow or too busy to handle the request. 3. BE Logs (
<http://be.INFO|be.INFO>
) When the I/O adaptor makes a decision to bypass the disk, it doesn't usually log every single skip (to avoid flooding logs), but it will log status changes or configuration alerts. You can search your

<http://be.INFO|be.INFO>

logs for: * io_adaptor
: Look for strings like

io_adaptor skip read

disk is too busy

. * block_cache
: General cache health messages. 4. Tracking via Disks You can monitor the physical disk performance directly on the OS to see if they are the bottleneck: * iostat -xz 1
: Look at the

%util

and

await

columns for the drives used by

storage_root_path

. * If await
(average latency) for your cache disks consistently stays high (>10-20ms for SSDs), the StarRocks I/O adaptor will automatically start favoring remote S3/HDFS reads because the network is effectively "faster" than your overloaded local disk. How to tune it If you feel the adaptor is being too aggressive (i.e., skipping the disk even when it's only slightly busy), you can adjust the BE configuration: * datacache_skip_read_factor
: (Default is usually adaptive). This controls how much I/O traffic is diverted. * Disable it (For Testing): If you want to force the system to always use the cache disk regardless of latency, run:

Copy code

sql
    SET GLOBAL enable_datacache_io_adaptor = false;

Warning: If your disks are actually slow, disabling this will cause your query latency to spike (tail latency). References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • data_source/data_cache_observe.md • administration/management/monitoring/metrics.md References • data_source/data_cache_observe.md

Philip Oliva

04/10/2026, 5:36 PM

@Rocky why? mysql> SHOW PROC '/cluster_balance/balance_stat'; Empty set (0.035 sec)

Rocky

04/10/2026, 5:36 PM

The most common reason for SHOW PROC '/cluster_balance/balance_stat'
returning an Empty set is that you are likely running StarRocks in Shared-data mode (v3.0+). Here is a detailed breakdown of why this happens and what it means for your cluster: 1. Shared-data Architecture (Most Likely) If your cluster is configured with shared-data (storing data on S3, OSS, or HDFS), the concept of Replica Balancing does not apply. * How it works: In shared-data mode, data is centralized in remote storage. BE or CN nodes do not maintain their own persistent local replicas of the data; they simply cache it. * The Result: Since there are no local replicas to move or balance between nodes, the "Tablet Scheduler" (which handles balancing) has no stats to report. The path

/cluster_balance/balance_stat

exists in the FE's internal structure but contains no data. 2. The Balancer is Disabled If you are in a Shared-nothing (Standard/Local Disk) architecture, the balancer might be explicitly disabled via FE configuration. You can check this by running:

Copy code

sql
ADMIN SHOW FRONTEND CONFIG LIKE '%balance%';

disable_balance

is set to

true

, the FE will not calculate balance statistics, and the result may be empty. You can enable it temporarily with:

Copy code

sql
ADMIN SET FRONTEND CONFIG ("disable_balance" = "false");

3. Fresh or Single-Node Cluster * Single BE Node: If your cluster only has one BE node, balancing is impossible (there is nowhere to move data to). While it sometimes shows "Balanced: true" in older versions, in some newer builds, it may return an empty set if the scheduling logic determines there are no viable targets. * Initialization: If the FE has just been restarted, it may take a few minutes (governed by

tablet_sched_checker_interval_seconds

) for the Tablet Scheduler to complete its first scan and populate these statistics. 4. No Tablets Created If the cluster is completely empty (no databases or tables created yet), there are no tablets to track, repair, or balance. You can verify your tablet count with:

Copy code

sql
SHOW PROC '/statistic';

TabletNum

is 0, the balance stat will remain empty. How to verify your architecture If you aren't sure which mode you are in, run:

Copy code

sql
SHOW BACKENDS\G

* If the nodes are listed as

CN

(Compute Nodes) or if your table creation statements use

PROPERTIES("storage_volume" = "...")

, you are in Shared-data mode, and the "Empty set" for balance stats is expected and normal. Rocky's Tip: In Shared-data clusters, instead of looking at tablet balance, you should monitor CN/BE Resource usage (CPU/Memory) and Data Cache hit rates, as the system automatically handles the data distribution at the storage layer. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/resource_management/Replica.md

Philip Oliva

04/10/2026, 5:53 PM

"*1. Shared-data Architecture (Most Likely)* If your cluster is configured with shared-data (storing data on S3, OSS, or HDFS), the concept of Replica Balancing does not apply. * How it works: In shared-data mode, data is centralized in remote storage. BE or CN nodes do not maintain their own persistent local replicas of the data; they simply cache it. * The Result: Since there are no local replicas to move or balance between nodes, the "Tablet Scheduler" (which handles balancing) has no stats to report. The path

/cluster_balance/balance_stat

exists in the FE's internal structure but contains no data." If above is true why would be_tablets table show which CN IDs own which tablets. ???? SELECT bt.BE_ID AS cn_id, tc.TABLE_SCHEMA, tc.TABLE_NAME, bt.TABLE_ID, bt.TABLET_ID, bt.DATA_SIZE FROM information_schema.be_tablets AS bt JOIN information_schema.tables_config AS tc ON bt.TABLE_ID = tc.TABLE_ID;

Philip Oliva

04/10/2026, 5:59 PM

@Rocky "*1. Shared-data Architecture (Most Likely)* If your cluster is configured with shared-data (storing data on S3, OSS, or HDFS), the concept of Replica Balancing does not apply. * How it works: In shared-data mode, data is centralized in remote storage. BE or CN nodes do not maintain their own persistent local replicas of the data; they simply cache it. * The Result: Since there are no local replicas to move or balance between nodes, the "Tablet Scheduler" (which handles balancing) has no stats to report. The path

/cluster_balance/balance_stat

Rocky

04/10/2026, 5:59 PM

This is an excellent catch and touches on the core architectural difference between Physical Storage and Logical Responsibility. The reason you see

BE_ID

(or

CN_ID

) mapping to tablets in

information_schema.be_tablets

while

balance_stat

remains empty is that StarRocks uses two different systems for "ownership." 1. "Physical Replicas" vs. "Logical Responsibility" * In Shared-nothing (Empty
balance_stat
context): Ownership means a BE has a physical copy of the data on its local hard drive. The "Balancer" (which populates

balance_stat

) is a background thread that moves these heavy files around to ensure no one's disk gets too full. * In Shared-data (Your cluster): There is no physical replica on the CN nodes; the "source of truth" is S3/HDFS. However, for a distributed system to work efficiently, the FE must still assign a "Leader" node for every tablet. 2. Why does a CN "own" a tablet in Shared-data mode? Even if data is in S3, the system needs to know which specific CN is responsible for: * Cache Locality (Consistent Hashing): To make the Data Cache effective, StarRocks ensures that Query A and Query B for "Tablet 100" always go to "CN Node 1." If every query for the same data went to random nodes, your cache hit rate would be 0%. * Compaction: Background tasks like merging small files into large ones (Compaction) must be performed by one specific node to avoid duplicate work and S3 write conflicts. * Metadata Management: The CN handles the metadata (versioning) for that tablet during ingestion. 3. Why is
balance_stat
empty then? The

SHOW PROC '/cluster_balance/balance_stat'

command is part of the Legacy Tablet Scheduler. * This scheduler is designed to look at Disk Capacity (%) and Replica Counts. * In Shared-data mode, there is no "Disk Capacity" to balance (since storage is infinite on S3) and no "Replica Count" (since there's only 1 logical tablet). * Because the criteria for the legacy balancer (Disk usage vs. Replica count) don't exist in shared-data, the internal

balance_stat

table has nothing to report. 4. How the "Ownership" you see in
be_tablets
is decided In your Shared-data cluster, the mapping you see in your query is decided by a Deterministic Mapping (often based on Consistent Hashing): 1. When a CN node joins the cluster, the FE calculates which tablets it should be "responsible" for. 2. If you add 10 more CN nodes, the FE will "re-map" some of those tablets to the new nodes. 3. Crucially: Unlike Shared-nothing, this "re-mapping" is near-instant because no data actually moves—the new CN node just starts reading from S3 and building its own local cache from scratch. Summary * be_tablets
: Shows which node is currently the Compute & Cache Master for a tablet. * balance_stat
: Shows the status of Physical File Migration between disks (which doesn't happen in shared-data mode). Your SQL query is the correct way to see the current distribution of compute/cache responsibility across your CN nodes! References • sql-reference/information_schema/be_tablets.md • administration/management/resource_management/Replica.md • introduction/Architecture.md • faq/shared_data_faq.md • administration/management/monitoring/metrics.md References • administration/management/resource_management/Replica.md • faq/shared_data_faq.md

Rocky

04/10/2026, 5:59 PM

• deployment/shared_data/feature-support-shared-data.md • table_design/data_distribution/Data_distribution.md

Philip Oliva

04/10/2026, 10:36 PM

@Rocky which parameter was one that is interval to check is CNs need rebalancing?

Rocky

04/10/2026, 10:36 PM

The parameter you are looking for that controls how often the FE checks if the nodes (both BEs and CNs) need rebalancing is: tablet_sched_checker_interval_seconds
* Default:

(seconds) * Description: This is the primary "heartbeat" of the Tablet Scheduler in the FE. Every 20 seconds, the scheduler wakes up to scan the cluster for two things: 1. Repairs: Tablets that are missing replicas or have version mismatches. 2. Balance: Tablets that need to be moved from high-load nodes to low-load nodes (or newly added nodes). ────────── Important Context for CNs (Shared-data Mode) While the scheduler checks every 20 seconds, its decisions are influenced by two other critical parameters: 1. tablet_stat_update_interval_second
(Default:

/ 5 minutes): Even though the checker runs every 20 seconds, the statistics (like tablet counts and node load) that it uses to make balancing decisions are only refreshed every 5 minutes. If you add a new CN, it might take a few minutes for the FE to "see" the updated stats and begin the rebalancing process. 2. tablet_sched_be_down_tolerate_time_s
(Default:

/ 15 minutes): If a CN goes offline, the scheduler doesn't immediately rebalance its tablets to other nodes. It waits for this "tolerate time" (15 minutes) to avoid unnecessary data movement in case the node is just restarting or experiencing a brief network blip. 3. lake_enable_balance_tablets_between_workers
(Default:

true

): Specifically for shared-data clusters (v3.3.4+), this parameter must be

true

for the scheduler to automatically migrate tablet "responsibility" between Compute Nodes to ensure an even distribution. How to check or change it You can check the current value in your cluster by running:

Copy code

sql
ADMIN SHOW FRONTEND CONFIG LIKE 'tablet_sched_checker_interval_seconds';

To change it temporarily (until next FE restart):

Copy code

sql
ADMIN SET FRONTEND CONFIG ("tablet_sched_checker_interval_seconds" = "10");

Rocky's Tip: Decreasing this interval makes the cluster respond faster to changes (like adding a node), but it increases the background CPU overhead on the Leader FE. For most production clusters, the default 20s is optimal. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md

Philip Oliva

04/10/2026, 10:38 PM

@Rocky if I change

tablet_sched_checker_interval_seconds

to 2 or 5 minutes what is the main risk for scale in and scale out?

Rocky

04/10/2026, 10:39 PM

Changing

tablet_sched_checker_interval_seconds

from the default 20 seconds to 2 or 5 minutes significantly slows down the "brain" of the cluster. While it reduces the CPU load on the FE Leader, it introduces several operational risks for scale-in and scale-out operations. Here are the main risks: 1. Extended Scale-In (Decommissioning) Time When you scale in a cluster using

DECOMMISSION

, the FE must move tablets off the leaving node and verify they are safely recreated elsewhere. * The Risk: Decommissioning is not a single-step process. The Tablet Scheduler often schedules tablets in batches (limited by

max_scheduling_tablets

). If your interval is 5 minutes, the FE only checks the progress and schedules the next batch once every 5 minutes. * Result: A process that might normally take 30 minutes could take several hours, keeping your old nodes online and costing money/resources longer than necessary. 2. Slow "Warm-up" during Scale-Out In both Shared-nothing and Shared-data modes, the goal of scaling out is to offload work to new nodes as fast as possible. * The Risk: New nodes will sit idle for up to 5 minutes before the FE even considers giving them work. * Shared-data Impact: In shared-data mode, rebalancing is just a metadata change (reassigning ownership). With a 5-minute interval, you might see existing CN nodes struggling with high CPU/Memory for several minutes after a scale-out because the "ownership" hasn't shifted to the new, empty nodes yet. 3. Increased "Vulnerability Window" (HA Risk) The Tablet Checker is responsible for identifying tablets that have fewer than the required number of replicas (e.g., if a BE dies). * The Risk: If a node fails during a scale-in/out operation, your data's health check is delayed by your custom interval. * Result: If you are in the middle of scaling in and another node happens to fail, you have a much longer window (up to 5 minutes +

tablet_sched_be_down_tolerate_time_s

) where your data might be under-replicated and at risk of loss or unavailability before the system even begins to repair it. 4. Delayed Load Balancing (Hotspots) The Tablet Scheduler uses the

ClusterLoadStatistics

(which updates every 1 minute) to decide where to move data. * The Risk: If the checker only runs every 5 minutes, it is using "old" load statistics and reacting very slowly to traffic spikes. * Result: If a specific node becomes a "hotspot" (due to a heavy query or high ingestion), the cluster will be much slower to move tablet responsibility away from that node, leading to prolonged query latency for users. Rocky’s Recommendation: * Scaling In/Out: If you are actively performing a maintenance window or scaling, keep it at the default 20s. You want the cluster to be as reactive as possible. * When to increase it: Only increase this value (to perhaps 60s, rarely 5m) if you have an extremely large cluster (e.g., 500+ nodes and millions of tablets) and you are seeing the FE Leader's CPU pegged at 100% specifically by the

TabletChecker

thread. To safely check your current status before changing:

Copy code

sql
-- Check how many tablets are currently being scheduled/balanced
SHOW PROC '/cluster_balance/running_tablets';

References • administration/management/FE_configuration.md • administration/management/resource_management/Replica.md • deployment/shared_data/feature-support-shared-data.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References

Rocky

04/10/2026, 10:39 PM

• administration/management/FE_configuration.md • administration/management/BE_configuration.md

Philip Oliva

04/10/2026, 10:43 PM

@Rocky I normally see deadline exceeded errors about 20 seconds after new CN joins. I know the errors happen around rebalancing on shared-data cluster. I thought maybe if rebalancing only happened once every 2 minutes it would give time for FEs StarOS manager to catch up.

Open in Slack

Previous Next