@Rocky My datacache disk usage is only about ~20% cluster wide. Why for this metric sum(rate(fslib_read_io_size_sum{fstype="cachefs",starrocks_cluster="$cluster"}[$interval])) * 100 / sum(rate(fslib_read_io_size_sum{starrocks_cluster="$cluster"}[$interval])) It never hits 100%. Sometimes even drops to 92% temporarily but mostly hovers around 98-99%

It is normal for the Data Cache hit ratio to hover around 98–99% and not reach a perfect 100%, even when you have plenty of disk space. Based on the StarRocks documentation, there are several architectural reasons for this behavior: 1. Asynchronous Population (Default) By default, StarRocks uses asynchronous population (

enable_datacache_async_populate_mode = true

). * Why it affects the ratio: When data is first accessed, the system fetches it from remote storage (S3/HDFS) and returns it to the user immediately. The process of writing that data into the local cache happens in the background. * Result: The very first time any piece of data is read, it counts as a "miss" (remote read). Even if you query the same dataset repeatedly, small background adjustments or new data blocks being accessed will result in occasional remote reads. 2. I/O Adaptation StarRocks has an I/O adapter enabled by default. * Why it affects the ratio: If the local cache disk is experiencing high latency or heavy load, the system may decide to fetch data directly from remote storage to avoid bottlenecking the query on a slow local disk. * Tuning: You can disable this by setting

enable_datacache_io_adapter = false

if you want to force the system to prefer the cache regardless of disk pressure. 3. Metadata and Internal Small Reads The metric

fslib_read_io_size_sum

tracks total I/O throughput. * Some metadata operations or specialized file headers might not always be cached in the block cache (which typically handles larger data blocks). * If your workload involves scanning many small files or frequently changing partitions, there will always be a baseline of "discovery" I/O that must go to the source. 4. Cache Population Rules Depending on your version and configuration, not every read is "eligible" to be cached. * Check: You can run

EXPLAIN VERBOSE <query>

and look for

dataCacheOptions={populate: true/false}

. * If it is

false

, that specific read will never be cached. You can change the global behavior by setting

populate_datacache_mode = always

. Why Disk Usage stays at 20% The Disk Usage and Hit Ratio are largely decoupled until you hit the capacity limit: * Low Usage: Your "working set" (the data actually being queried) currently fits into ~20% of your allocated cache space. * Eviction: StarRocks only begins evicting data once it hits the threshold (usually 80-90%). Since you are at 20%, the system is not deleting anything; the <100% hit rate is simply due to the "cold" or "asynchronous" nature of the reads mentioned above. Recommendations to Improve the Ratio If you want to push the hit ratio closer to 100%, you can try these session or global variables: *

SET GLOBAL enable_datacache_async_populate_mode = false;

(Makes population synchronous; first-time queries will be slower but will populate the cache immediately). *

SET GLOBAL enable_datacache_io_adapter = false;

(Prevents bypassing the cache during high disk I/O). * Use

CACHE SELECT ...

(StarRocks 3.3+) to explicitly warm up the cache for specific tables/partitions. References • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md