@Rocky I'm using 4.0.8 shared data starrocks, I have a configured pvc for cache of 100GB, out of which only 70GB is disk cache and 10GB page cache(mem). If I run query which scans more than data to be cached by compute node, I see data cache is not written to disk(generally after data read and cache enabled for table, data written to cache disk for further). why is that happening? If this happens does it cause any problem to cluster?

In StarRocks Shared Data (including version 4.0.x), the behavior you are observing is likely due to the Data Cache Admission Policy and Cache Thrashing Protection. Why is data not written to the cache? When you run a query that scans significantly more data than the allocated cache capacity (in your case, scanning > 70GB with a 70GB disk cache), StarRocks employs several protective mechanisms: 1. Cache Thrashing Protection: If the system allowed a single massive scan to fill the cache, it would evict all existing "hot" data (LRU/SLRU). Because the scan size exceeds the total cache size, the data being written would itself be evicted before it could ever be reused. To prevent this "thrashing," StarRocks may bypass the cache write for queries that exceed a certain percentage of the total cache quota. 2. Adaptive Admission Policy: In newer versions (like 4.0), the Data Cache (formerly Block Cache) has an adaptive mechanism. If the system detects a "large scan" or a "one-off" full-table scan that doesn't fit into the cache medium, it prioritizes system stability and avoids the overhead of writing that data to disk. 3. Table Properties (

partition_duration

): Check if your table has

datacache.partition_duration

set. If the data you are scanning falls outside this "hot" window, StarRocks will intentionally skip caching it to save space for more recent data. 4. Disk Safety Watermark: By default, StarRocks starts cache eviction and may restrict new writes when disk usage hits a "safe level" (typically 80% of the allocated quota). If your disk cache is already near this limit, new large writes might be skipped. Does this cause problems to the cluster? No, this is generally a beneficial behavior. * Prevents I/O Spikes: Writing 70GB+ of data to a PVC during a single query creates massive disk I/O and CPU overhead. If the data won't stay in the cache anyway, skipping the write saves resources. * Protects Hot Data: It ensures that smaller, frequently accessed "hot" datasets (like dimension tables or recent partitions) are not kicked out by a single giant report or batch job. * Stability: It prevents the "No space left on device" errors that can occur if multiple CN nodes attempt to fill shared or constrained PVCs simultaneously. How to verify and adjust * Check the Profile: Look at the

DataCacheWriteBytes

metric in your Query Profile. If it is 0 or very low despite a large

BytesRead

, the admission policy has bypassed the cache. * Force Caching (Warmup): If you know you need this data cached for subsequent queries, use the manual warmup feature introduced in v3.3+:

sql
    CACHE SELECT * FROM your_table WHERE ...;

This proactively loads data into the cache regardless of the standard query admission logic. * Increase Quota: If you frequently scan 100GB+ and want it cached, you must increase your PVC size and the

datacache_disk_size

(or

starlet_star_cache_disk_size_percent

). It is recommended to have a cache size at least 1.2x to 1.5x larger than your most frequent "hot" data working set. * Check Configurations: Verify these BE/CN configs: * `datacache_disk_size`: Ensure it matches your expected 70GB. * `starlet_star_cache_disk_size_percent`: (Default 80) Ensure it isn't capping your cache lower than expected. Summary: Your cluster is likely healthy. The system is protecting itself from "cache pollution" by a scan that is too large to be effectively cached in the large for the current cache configuration. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • using_starrocks/caching/query_cache.md

@Rocky I see a compute node scanning so much of data (500GB from bucket) but my cache size is 100GB and cpu percent is almost near 80% and I see 1,2 other compute nodes cpu also near 70% but not scanning so much of data(maybe not scheduled to scan more data from bucket). in which cases, this can happen? even though cpu available in other nodes and tablets are good enough more than compute nodes in a table, I see scan is very less in other compute nodes but not this cn0 compute node and until cn0 is deleted, queries entering same node causing too slow (as scanning so much data from bucket) and queries timed out, when this can happen? and how to fix?

The high imbalance on

cn0

in your v4.0.8 shared-data cluster is likely caused by Tablet-to-Node Affinity (Cache-Aware Scheduling) combined with Data Skew. Why is this happening? 1. Cache-Aware Scheduling: In shared-data mode, StarRocks uses a Consistent Hashing algorithm to bind specific tablets to specific Compute Nodes. This is intentional: by routing scan tasks for the same tablet to the same node, StarRocks maximizes the local Data Cache hit rate. 2. Tablet Skew: If your table has a few massive tablets (e.g., if you see "500GB" coming from a very small number of tablets), those tablets are assigned to

cn0

by the hash. Even if

cn1

and

cn2

have idle CPU, they won't pick up those scan tasks because they don't "own" those tablets in the cache mapping. 3. Single Tablet Bottleneck: If your query scans 500GB but the data is concentrated in a single tablet, only one node can process that tablet at a time. This leads to the "overcrowded" behavior where new queries hitting

cn0

are queued behind the massive scan. How to Fix 1. Re-Bucketing (Long-term Solution) For 500GB of data, you should ensure the data is split into many smaller tablets. * Target Tablet Size: Aim for 1GB to 5GB per tablet. If you have 500GB, you should have at least 100–200 buckets (tablets). * Check Bucketing Key: Ensure your

DISTRIBUTED BY HASH(column)

uses a high-cardinality column. If you bucket by a column with few unique values, one tablet will grow disproportionately large (skew). * Action: Use

ALTER TABLE ... SET ("bucket_size" = "...")

or recreate the table with more buckets to spread the load across all CNs. 2. Enable Internal Parallelism (Session Level) You can force StarRocks to use more threads to scan a single tablet, which may help

cn0

finish faster, though it won't move the work to other nodes:

sql
SET enable_tablet_internal_parallel = true;

3. Adjust Cache Affinity (Advanced) If you prefer load balancing over cache hits for a specific large query, you can try to disable the preference for nodes with cache (though this is usually managed internally by the scheduler): * Check if

lake_enable_balance_tablets_between_workers

is set to

true

in your FE configuration. This allows the FE to migrate tablet assignments if a node becomes a bottleneck. 4. Protect the Cluster with Resource Groups To prevent one giant query from causing timeouts for everyone else on

cn0

, use Resource Groups to cap the CPU and memory a single query or user can consume:

sql
CREATE RESOURCE GROUP big_scan_group 
WITH (
    "cpu_core_limit" = "50%", 
    "mem_limit" = "30%",
    "big_query_cpu_second_limit" = "100" -- Kills queries exceeding this CPU time
);

Verification Run

EXPLAIN ANALYZE <your_query>

and check the

OlapScanNode

section. Look for: * TabletCount: If this number is very small (e.g., 1 or 2) for a 500GB scan, you have a bucketing problem. * MorselsCount: If this is low, the work cannot be parallelized effectively. References • using_starrocks/Cost_based_optimizer.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md