@Rocky I could see for some queries in starrocks_audit_tbl__ , the query ran till timeout(60seconds) but scanBytes=0 and scanRows=0, cpuCostNs=0 . Is it not populating due to some issue with cluster configs? I have shared data with 4.0.6 version

In StarRocks, finding

scanBytes=0

,

scanRows=0

, and

cpuCostNs=0

for a query that timed out (e.g., at 60 seconds) typically indicates that the query never reached the execution phase on the Backend (BE) or Compute Node (CN). Instead of a cluster configuration issue with the metrics themselves, this usually points to the query being stuck in one of the following pre-execution stages: 1. Query Queuing (Most Likely) If you have Resource Groups or Query Queues enabled, a query may sit in a queue waiting for available slots. if it stays there until the

query_timeout

(or

query_queue_pending_timeout_second

) is reached, it will be cancelled without ever scanning data. * How to verify: Check the

pendingTimeMs

column in your

starrocks_audit_tbl__

. If this value is close to 60,000 ms, the query timed out while waiting to start. 2. Optimization/Planning Timeout The Frontend (FE) must plan the query (analyze, optimize, and distribute fragments) before execution starts. For complex queries (e.g., many joins), the optimizer might take a long time. * How to verify: Check the

planCpuCosts

column in the audit table. While this is in nanoseconds, a very high value suggests the FE spent the entire time planning. Also, check if the

state

is

ERR

with an error message like "planner optimize timeout". 3. Metadata or Lock Contention (Shared-Data Specific) In a Shared-data (Cloud-native) environment, the FE must fetch tablet metadata and locations from the internal StarMgr service or object storage. * If the FE encounters heavy lock contention or cannot communicate with the metadata service, the query can hang before any execution fragments are dispatched to the CNs. * In version 4.0.x, ensure your CNs are healthy and that there isn't a "Deadline Exceeded" or gRPC timeout occurring during the metadata lookup phase (check

fe.log

). 4. Metadata-Only Queries If the query is something like

SELECT count(*) FROM table

where the result can be served entirely from the FE's metadata cache (though rare for a 60s timeout), or if it is a DDL/administrative command, it will naturally show 0 scan metrics. Recommended Next Steps: 1. Check `pendingTimeMs`: Run

SELECT queryId, pendingTimeMs, planCpuCosts, state, errorCode FROM starrocks_audit_db*.starrocks_audit_tbl* WHERE queryId = 'your_query_id';

. 2. Examine FE Logs: Search

fe.log

for the specific

queryId

to see if there are "Failed to get partitions" or "RPC timeout" errors during the planning phase. 3. Check Resource Group usage: If

pendingTimeMs

is high, you may need to increase the concurrency limits of your resource groups. References • best_practices/query_tuning/query_profile_operator_metrics.md • administration/management/audit_loader.md • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • sql-reference/System_variable.md

@Rocky here are values of audit table for that query planCpuCosts 5,031,233,981.86 planMemCosts 529,588,079.11 pendingTimeMs -1 candidateMVs null hitMvs null state ERR errorCode TIMEOUT queryTime 60,057 scanBytes 0 scanRows 0 returnRows 0 cpuCostNs 0 memCostBytes 0 stmtId 783227 isQuery 1

Based on the audit data provided, the fact that your query timed out after 60 seconds with zero scan metrics (

scanBytes=0

,

scanRows=0

,

cpuCostNs=0

) but had a low planning CPU cost (

planCpuCosts

≈ 5s) indicates that the query was stuck in a waiting state on the Frontend (FE) before it could even begin execution on the Compute Nodes (CN). Here is the breakdown of why this happens in StarRocks 4.0.x (Shared-data): 1. The "Missing" 55 Seconds * Total Query Time: ~60.05 seconds. * Planning CPU: ~5.03 seconds. * Execution Metrics: 0 (The query never reached the CNs). * The Gap: There are roughly 55 seconds where the FE was neither "calculating" the plan nor "executing" it. This is almost always wait time. 2. Why

pendingTimeMs

is

-1

In the StarRocks audit table, a

pendingTimeMs

of

-1

typically means the query bypassed the Resource Group queue. It was not waiting for a "slot" in a concurrency-limited resource group. Instead, it was blocked by internal system waits. 3. Likely Causes in Shared-Data (4.0.6) In a shared-data architecture, the FE must perform several synchronous operations before dispatching a query. If any of these are slow or blocked, the query will hang at 0% progress: * Metadata Lock Contention: The FE uses global locks to manage metadata. If another long-running operation (like a massive

ALTER TABLE

or a heavy metadata synchronization) is holding a lock, your query will wait until it times out. * StarMgr / Remote Metadata Latency: In version 4.x, the FE frequently communicates with the internal metadata service (StarMgr) to fetch tablet and file locations from object storage (S3/OSS). If the communication with StarMgr is slow or the object storage metadata is slow to respond, the FE will "hang" waiting for these network responses. This wait time is not counted as

planCpuCosts

(which only measures CPU-intensive optimizer logic). * Fragment Dispatch Hang: After the plan is generated (the 5s part), the FE tries to send "fragments" to the CNs via RPC. If the CNs are overloaded, unresponsive, or experiencing network issues, the FE may hang here. If the RPC never succeeds, no execution starts, and

scanBytes

remains 0. Troubleshooting Steps: 1. Check FE Logs (

fe.log

): Search for the specific

queryId

. Look for warnings such as: *

get tablet location failed

*

lock wait timeout

*

StarMgr RPC timeout

*

failed to dispatch query

2. Analyze Metadata Scale: Does this query involve a table with a massive number of partitions or tablets? Overwhelming the FE with metadata lookups is a common cause for these "pre-execution" hangs. 3. Check Catalog Health: If you are using an External Catalog (e.g., Iceberg/Hive), the FE might be waiting for the remote Metastore. 4. StarMgr Status: In shared-data mode, ensure the StarMgr service is healthy and not reporting high latency or "Deadline Exceeded" errors. If this happens frequently for specific queries, it often points to metadata lock contention or network latency between the FE and the metadata store/CNs3 service/object storage. References • administration/management/audit_loader.md • administration/management/query_queues.md • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • best_practices/query_tuning/query_profile_operator_metrics.md References • administration/management/audit_loader.md

@Rocky if my cluster have 2compute nodes only and I run query which many partitions and after sclaed to 7nodes, tablets will be redistributed. If I run same query with 7nodes, does it tries to hit first 1 or 2nodes(as data gets cached there) or as tablets are redistributed to all nodes, it will try hit all nodes?

In StarRocks Shared-data (Cloud-native) mode, the relationship between data and compute nodes is logical, not physical. Here is exactly what happens when you scale from 2 nodes to 7 nodes: 1. No Physical Redistribution Unlike the Shared-nothing architecture (with BEs), there is no background data rebalancing (no moving data from one disk to another). All data stays in your object storage (S3/OSS/HDFS). Only the tablet-to-node mapping in the Frontend (FE) metadata changes. 2. Consistent Hashing for Cache Locality StarRocks uses Consistent Hashing to assign tablets to Compute Nodes (CNs). This is designed to maximize the "Data Cache" hit rate: * Original 2 Nodes: When you scale out, consistent hashing ensures that a portion of the tablets (roughly $2/7 \approx 28\%$) remains assigned to the original two nodes. For these tablets, the query will hit the local cache as before. * New 5 Nodes: The remaining tablets (roughly $5/7 \approx 72\%$) will be re-mapped to the five new nodes. Since these nodes have empty local disks, the first time they process these tablets, they will experience a cache miss, pull the data from object storage, and then cache it locally. 3. Will the query hit all nodes? Yes. If your query scans many partitions or a large amount of data, the FE will distribute the query fragments across all 7 nodes. * It does not try to "hit the first 2 nodes" just because they have the cache. * Instead, it prioritizes parallelism. By using all 7 nodes, you get $3.5\times$ the CPU and Memory power. * Even though the new nodes have a "cold" cache on the first run, the increased CPU count usually makes the query faster overall, and subsequent runs will be significantly faster once the 5 new nodes populate their caches. Summary of Behavior: Feature | Behavior after Scaling (2 $\rightarrow$ 7 nodes) Workload Distribution | Distributed across all 7 nodes immediately. Cache Hit (Old Nodes) | High. They still own $\sim$28% of their original tablets. Cache Hit (New Nodes) | Low (Miss) on 1st run; High on 2nd run. Data Movement | Zero. Only the metadata "assignment" changes. Note on your previous "Timeout" issue: If your query was timing out with

scanBytes=0

, scaling to 7 nodes might help if the issue was a lack of CPU/Memory for planning or internal metadata processing. However, if the issue was metadata lock contention or FE-to-Object-Storage latency, adding more CNs may not solve it. Monitor the

pendingTimeMs

and

fe.log

to see if the FE is struggling to dispatch the query to the new nodes. References • introduction/Architecture.md • deployment/shared_data/feature-support-shared-data.md • administration/management/FE_configuration.md • using_starrocks/caching/query_cache.md • best_practices/bucketing.md References • best_practices/bucketing.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/Colocate_join.md • deployment/shared_data/s3.md