http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> How much data cache disk size shoul...
# questions-and-troubleshooting
b
@Rocky How much data cache disk size should I consider for shared data modes? What is a good way to measure a balance of data cache disk size?
✅ 1
r
In StarRocks shared-data mode, the Data Cache is a critical component for achieving high performance by caching remote data (from S3, HDFS, etc.) onto local disks (BE or CN nodes). 1. How much Data Cache disk size to consider? There is no "one-size-fits-all" number, but a common approach is to size the cache based on your "Hot Data" volume. * Rule of Thumb: A common starting point is to allocate enough disk space to cover 20% to 50% of your total data volume, or ensure it can fit the daily active dataset (data accessed within the last 24–48 hours). * Default Configuration: By default, StarRocks allocates 80% of the disk space specified in 
storage_root_path
for the Data Cache. This is controlled by the BE configuration 
starlet_star_cache_disk_size_percent
. * Minimum Recommendation: If your hot data is small, even a few hundred GBs per node can significantly boost performance. If your hot data is massive (e.g., several TBs), you should consider using high-performance NVMe SSDs to handle the I/O throughput. 2. How to measure the balance? The goal is to find a "sweet spot" where your cache hit ratio is high enough to satisfy performance requirements without over-investing in expensive local storage. Key Metrics to Monitor You can use the StarRocks BE API to check cache efficiency: 
http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/app_stat
Metric | Goal | Meaning 
block_cache_hit_rate
|
80%
 | A high hit rate indicates most queries are served from local disk. 
block_cache_disk_used_rate
| ~80% | The cache naturally fills up to its limit (default 80%) and then evicts data. 
fslib read io_latency
| Low (ms) | Measures the latency of reading from the cache. High latency suggests disk I/O saturation. Balancing Strategies 1. Identify Thrashing: If 
block_cache_hit_rate
is consistently low (e.g., < 50%) while 
block_cache_disk_used_rate
is at its limit, your cache is likely "thrashing" (new data constantly evicting useful data). You should increase the disk size. 2. Diminishing Returns: If increasing the cache size from 1TB to 2TB only improves the hit rate by 1-2%, you have likely reached the point of diminishing returns. 3. Storage Medium: If hit rates are high but query latency is still high, the bottleneck might be the disk speed rather than the size. Switching from HDD to NVMe SSD or increasing the number of disks to share I/O pressure is recommended. 4. Warmup Usage: Use the 
CACHE SELECT
command (introduced in v3.3.0) to proactively load important tables into the cache, ensuring high hit rates for critical reports regardless of the overall cache size balance. Configuration Parameters * 
starlet_star_cache_disk_size_percent
: The percentage of disk capacity Data Cache can use (Default: 
80
). * 
datacache_disk_size
: A fixed size (e.g., 
2T
) for the cache. The system takes the greater of this and the percentage setting. * 
datacache.partition_duration
: (Table property) Can be used to prioritize caching for more recent data partitions. References • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md • data_source/data_cache.md • deployment/shared_data/feature-support-shared-data.md
b
@Rocky Here are some example stats from the compute nodes i have
Copy code
{
    "block_cache_hit_bytes": 0,
    "block_cache_miss_bytes": 2010129650706,
    "block_cache_hit_rate": 0.0,
    "block_cache_hit_bytes_last_minute": 0,
    "block_cache_miss_bytes_last_minute": 2622601322,
    "block_cache_hit_rate_last_minute": 0.0,
    "page_cache_hit_count": 5428897,
    "page_cache_miss_count": 40393782,
    "page_cache_hit_rate": 0.12,
    "page_cache_hit_count_last_minute": 15492,
    "page_cache_miss_count_last_minute": 42274,
    "page_cache_hit_rate_last_minute": 0.27
}

{
    "block_cache_hit_bytes": 0,
    "block_cache_miss_bytes": 2054641467293,
    "block_cache_hit_rate": 0.0,
    "block_cache_hit_bytes_last_minute": 0,
    "block_cache_miss_bytes_last_minute": 2295079659,
    "block_cache_hit_rate_last_minute": 0.0,
    "page_cache_hit_count": 5674182,
    "page_cache_miss_count": 41121389,
    "page_cache_hit_rate": 0.12,
    "page_cache_hit_count_last_minute": 16361,
    "page_cache_miss_count_last_minute": 39938,
    "page_cache_hit_rate_last_minute": 0.29
}

{
    "block_cache_hit_bytes": 0,
    "block_cache_miss_bytes": 2165121857589,
    "block_cache_hit_rate": 0.0,
    "block_cache_hit_bytes_last_minute": 0,
    "block_cache_miss_bytes_last_minute": 3262243755,
    "block_cache_hit_rate_last_minute": 0.0,
    "page_cache_hit_count": 6170005,
    "page_cache_miss_count": 42928742,
    "page_cache_hit_rate": 0.13,
    "page_cache_hit_count_last_minute": 20356,
    "page_cache_miss_count_last_minute": 48063,
    "page_cache_hit_rate_last_minute": 0.3
}
This does not look great as the block cache is almost not being used?
r
It definitely looks like your Block Cache (part of the StarRocks Data Cache) is not being utilized effectively, as indicated by the 0% hit rate and 0 hit bytes despite high miss bytes. In StarRocks, the Block Cache typically refers to the disk-based cache used in shared-data clusters (Cloud-native) or when querying external catalogs (Hive, Iceberg, etc.). Here is a checklist to troubleshoot why your cache isn't being populated: 1. Check for Cache Population Rules (SR v3.3.2+) Starting from StarRocks v3.3.2, there are specific rules to prevent "cache pollution" from large one-off scans. The system will not populate the block cache if: * Full Table Scans: You are scanning all columns of a table (e.g., 
SELECT *
). * Full Partition Scans: Your query does not include partition filters and scans all partitions. * Non-Select Statements: Commands like 
INSERT INTO SELECT
or 
ANALYZE TABLE
do not populate the cache. Solution: Try running a query with a specific 
WHERE
clause on a partition key and selecting only a few columns to see if 
block_cache_hit_bytes
starts to increase. 2. Verify CN Configuration The Block Cache requires local disk space to be explicitly assigned on your Compute Nodes (CN). Check your 
cn.conf
for the following: * 
storage_root_path
: This must be set to a valid local directory where the cache files will be stored. * 
datacache_enable
: Ensure this is 
true
(default is true). * 
datacache_disk_size
: Ensure this is set to a value greater than 0 (e.g., 
214748364800
for 200GB). If this is not set, the cache may not have space to write. You can verify these at runtime by running:
Copy code
sql
SELECT * FROM information_schema.be_configs 
WHERE NAME LIKE "%storage_root_path%" 
   OR NAME LIKE "%datacache_disk_size%"
   OR NAME LIKE "%datacache_enable%";
3. Check the Data Cache Status API You can get more granular details (like whether writes are failing) by hitting the debug endpoint on one of your CN nodes:
Copy code
bash
curl http://<CN_IP>:<CN_HTTP_PORT>/api/datacache/stat
Look for: * `block_cache_status`: It should be 
NORMAL
. * `block_cache_write_fail_count`: If this is high, it indicates the system is trying to cache data but failing (likely due to disk permissions or lack of space). * `block_cache_disk_used_bytes`: Check if this is growing at all. 4. Session Variables Ensure that the session variable 
enable_scan_datacache
is set to 
true
(it is by default). You can check it with:
Copy code
sql
SHOW VARIABLES LIKE '%enable_scan_datacache%';
5. Query Profile Analysis To confirm exactly what is happening during a query, run a query and check its Profile:
Copy code
sql
-- Run your query
SELECT col FROM your_table WHERE partition_col = 'xxx';
-- Get the profile ID
get_last_query_info();
Look for the 
DataCacheReadBytes
and 
DataCacheWriteBytes
metrics in the profile. If 
DataCacheWriteBytes
is 0, it confirms the query is bypassing the cache population logic. Rocky's Tip: If you are on a version >= 3.3.2 and need to force population for a full scan, you might need to adjust your query patterns or check if specific session variables like 
datacache_populate_mode
(if available in your version) are restricting population. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md
b
@Rocky Some info
Copy code
SELECT * FROM information_schema.be_configs WHERE NAME LIKE "%storage_root_path%"    OR NAME LIKE "%datacache_disk_size%"   OR NAME LIKE "%datacache_enable%";
+-------+---------------------+---------------------------+--------+---------------------------+---------+
| BE_ID | NAME                | VALUE                     | TYPE   | DEFAULT                   | MUTABLE |
+-------+---------------------+---------------------------+--------+---------------------------+---------+
| 47034 | datacache_disk_size | 80%                       | string | 100%                      |       1 |
| 47034 | datacache_enable    | true                      | bool   | true                      |       0 |
| 47034 | storage_root_path   | /opt/starrocks/cn/storage | string | ${STARROCKS_HOME}/storage |       0 |
| 29058 | datacache_disk_size | 80%                       | string | 100%                      |       1 |
| 29058 | datacache_enable    | true                      | bool   | true                      |       0 |
| 29058 | storage_root_path   | /opt/starrocks/cn/storage | string | ${STARROCKS_HOME}/storage |       0 |
| 47014 | datacache_disk_size | 80%                       | string | 100%                      |       1 |
| 47014 | datacache_enable    | true                      | bool   | true                      |       0 |
| 47014 | storage_root_path   | /opt/starrocks/cn/storage | string | ${STARROCKS_HOME}/storage |       0 |
+-------+---------------------+---------------------------+--------+---------------------------+---------+
Datacache
Copy code
{
    "page_cache_mem_quota_bytes": 19713899888,
    "page_cache_mem_used_bytes": 19712283222,
    "page_cache_mem_used_rate": 1.0,
    "page_cache_hit_count": 5496031,
    "page_cache_miss_count": 40606211,
    "page_cache_hit_rate": 0.12,
    "page_cache_hit_count_last_minute": 8601,
    "page_cache_miss_count_last_minute": 28968,
    "page_cache_hit_rate_last_minute": 0.23,
    "block_cache_status": "NORMAL",
    "block_cache_disk_quota_bytes": 3457448673280,
    "block_cache_disk_used_bytes": 3457108934656,
    "block_cache_disk_used_rate": 1.0,
    "block_cache_disk_spaces": "/opt/starrocks/cn/storage/datacache:3457448673280",
    "block_cache_meta_used_bytes": 2143356235,
    "block_cache_hit_count": 145446986,
    "block_cache_miss_count": 1201127,
    "block_cache_hit_rate": 0.99,
    "block_cache_hit_bytes": 4639948636097,
    "block_cache_miss_bytes": 209996974364,
    "block_cache_hit_count_last_minute": 175969,
    "block_cache_miss_count_last_minute": 1377,
    "block_cache_hit_bytes_last_minute": 8614902679,
    "block_cache_miss_bytes_last_minute": 274681355,
    "block_cache_read_disk_bytes": 4639919169822,
    "block_cache_write_bytes": 1729893934440,
    "block_cache_write_success_count": 2130741,
    "block_cache_write_fail_count": 0,
    "block_cache_remove_bytes": 0,
    "block_cache_remove_success_count": 274100,
    "block_cache_remove_fail_count": 0,
    "block_cache_current_reading_count": 0,
    "block_cache_current_writing_count": 0,
    "block_cache_current_removing_count": 0
}

{
    "page_cache_mem_quota_bytes": 19713899888,
    "page_cache_mem_used_bytes": 19712728739,
    "page_cache_mem_used_rate": 1.0,
    "page_cache_hit_count": 5729122,
    "page_cache_miss_count": 41304319,
    "page_cache_hit_rate": 0.12,
    "page_cache_hit_count_last_minute": 7464,
    "page_cache_miss_count_last_minute": 28822,
    "page_cache_hit_rate_last_minute": 0.21,
    "block_cache_status": "NORMAL",
    "block_cache_disk_quota_bytes": 3489660928000,
    "block_cache_disk_used_bytes": 2428896018432,
    "block_cache_disk_used_rate": 0.7,
    "block_cache_disk_spaces": "/opt/starrocks/cn/storage/datacache:3489660928000",
    "block_cache_meta_used_bytes": 1531683800,
    "block_cache_hit_count": 146077600,
    "block_cache_miss_count": 1373841,
    "block_cache_hit_rate": 0.99,
    "block_cache_hit_bytes": 4624154839431,
    "block_cache_miss_bytes": 227551097778,
    "block_cache_hit_count_last_minute": 175167,
    "block_cache_miss_count_last_minute": 1101,
    "block_cache_hit_bytes_last_minute": 8601983006,
    "block_cache_miss_bytes_last_minute": 239384615,
    "block_cache_read_disk_bytes": 4624125595192,
    "block_cache_write_bytes": 1557336197699,
    "block_cache_write_success_count": 1941092,
    "block_cache_write_fail_count": 716,
    "block_cache_remove_bytes": 0,
    "block_cache_remove_success_count": 260878,
    "block_cache_remove_fail_count": 0,
    "block_cache_current_reading_count": 0,
    "block_cache_current_writing_count": 0,
    "block_cache_current_removing_count": 0
}

{
    "page_cache_mem_quota_bytes": 19713899888,
    "page_cache_mem_used_bytes": 19710451400,
    "page_cache_mem_used_rate": 1.0,
    "page_cache_hit_count": 6231185,
    "page_cache_miss_count": 43098843,
    "page_cache_hit_rate": 0.13,
    "page_cache_hit_count_last_minute": 8958,
    "page_cache_miss_count_last_minute": 31065,
    "page_cache_hit_rate_last_minute": 0.22,
    "block_cache_status": "NORMAL",
    "block_cache_disk_quota_bytes": 3457448673280,
    "block_cache_disk_used_bytes": 3457448148992,
    "block_cache_disk_used_rate": 1.0,
    "block_cache_disk_spaces": "/opt/starrocks/cn/storage/datacache:3457448673280",
    "block_cache_meta_used_bytes": 2140548296,
    "block_cache_hit_count": 147202574,
    "block_cache_miss_count": 1358228,
    "block_cache_hit_rate": 0.99,
    "block_cache_hit_bytes": 4643244119539,
    "block_cache_miss_bytes": 233765852135,
    "block_cache_hit_count_last_minute": 174778,
    "block_cache_miss_count_last_minute": 1245,
    "block_cache_hit_bytes_last_minute": 8605737752,
    "block_cache_miss_bytes_last_minute": 297297754,
    "block_cache_read_disk_bytes": 4643215418894,
    "block_cache_write_bytes": 1929037782116,
    "block_cache_write_success_count": 2297835,
    "block_cache_write_fail_count": 0,
    "block_cache_remove_bytes": 0,
    "block_cache_remove_success_count": 249429,
    "block_cache_remove_fail_count": 0,
    "block_cache_current_reading_count": 0,
    "block_cache_current_writing_count": 0,
    "block_cache_current_removing_count": 0
}
Copy code
SHOW VARIABLES LIKE '%enable_scan_datacache%';
+-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| enable_scan_datacache | true  |
+-----------------------+-------+
r
Hello! I'm Rocky. Based on the information you provided, here is an analysis of your StarRocks Data Cache status: Summary of Findings 1. Excellent Hit Rate: Your Block Cache hit rate is 99% across all nodes. This indicates that almost all data requests are being served directly from the local disk cache rather than remote storage, which significantly improves query performance. 2. Healthy Cache Saturation: BE nodes 47034 and 47014 have a 
block_cache_disk_used_rate
of 1.0 (100%). For a cache, being full is normal and expected behavior; it means the allocated 80% of your disk space is fully utilized, and StarRocks is using an LRU (Least Recently Used) policy to manage space. 3. Potential Issue on BE 29058: * This node is only at 70% usage, while others are at 100%. * Crucially, it shows 
block_cache_write_fail_count: 716
, whereas the other nodes have 0 failures. * Recommendation: Check the BE logs (
<http://be.INFO|be.INFO>
or 
be.WARNING
) for this specific node. This failure often indicates underlying disk I/O issues, permission problems, or the physical disk becoming full before the logical cache quota was reached. Metric Breakdown * Block Cache (Disk-based): * Hit Rate (99%): Exceptional. You are getting maximum value from your SSDs/disks. * Disk Quota: You have roughly 3.2 TiB allocated per node for caching (
block_cache_disk_quota_bytes
). * Page Cache (Memory-based): * Used Rate (1.0): Your memory-based page cache is also fully utilized. * Hit Rate (12-13%): This is lower than the block cache, which is common if your "hot" working set is larger than the available memory but fits well within the disk cache. * Configuration: * 
datacache_enable = true
and 
enable_scan_datacache = true
are correctly configured for the cluster and the session. Suggestions * Investigate BE 29058: Look for strings like 
DataCache
or 
st_write
in the logs of BE 29058 to understand why those 716 writes failed. * Monitor Disk Latency: Since your hit rate is so high, the performance of your queries is now heavily dependent on the I/O latency of the path 
/opt/starrocks/cn/storage/datacache
. Ensure these are high-performance SSDs (NVMe preferred) for the best results. Do you have a specific performance issue you are troubleshooting, or were you just checking the health of the cache? References • data_source/data_cache_observe.md • using_starrocks/caching/block_cache.md
b
@Rocky Just wondering if I can reduce the disk size for the cache
r
Yes, you can reduce the disk size for the Data Cache. Depending on whether you want to permanently change the limit or just clear the existing cache to reclaim space, there are a few ways to handle this. 1. Adjust the Cache Limit (Quota) You can modify the 
datacache_disk_size
configuration to set a specific limit (e.g., 
500G
, 
1T
). This can be done dynamically without a restart:
Copy code
sql
-- Adjust the disk limit for all BE nodes
UPDATE be_configs SET VALUE="500G" WHERE NAME="datacache_disk_size";

-- Or for a specific BE node
UPDATE be_configs SET VALUE="500G" WHERE NAME="datacache_disk_size" AND BE_ID=10005;
Note: Changes made via 
UPDATE be_configs
are not persistent. To make them permanent, you must also update the 
be.conf
or 
cn.conf
file on each node. 2. Physically Reclaiming Space A common point of confusion is that StarRocks does not always delete the physical files on disk immediately when data is evicted; it often marks that space as "overwritable" to maintain performance. If you see that your disk usage remains high even after reducing the quota, you can force a cleanup: * Option A: The "Zero Out" Trick (No restart required) Set the quota to 
0
to trigger a cleanup, and then set it back to your desired size:
Copy code
sql
    UPDATE be_configs SET VALUE="0" WHERE NAME="datacache_disk_size";
    -- Wait a moment for the system to process
    UPDATE be_configs SET VALUE="500G" WHERE NAME="datacache_disk_size";
* Option B: Manual Cleanup (Restart required) 1. Stop the BE/CN node. 2. Delete the contents of the 
datacache
directory (found under your 
storage_root_path
). 3. Restart the node. 3. Automatic Scaling If you prefer the system to manage this automatically based on disk pressure, you can enable auto-adjustment in your configuration:
Copy code
properties
datacache_auto_adjust_enable = true
With this enabled, the system will evict cache data when the disk usage reaches 
disk_high_level
(default 90%) and aim to keep it around 
disk_safe_level
(default 80%). Important Configurations to Check: * 
datacache_disk_size
: The absolute disk quota (e.g., 
200G
). * 
starlet_star_cache_disk_size_percent
: The percentage of the disk to use for cache (default is often 80%). The system usually takes the greater of these two values. References • data_source/data_cache_troubleshooting.md • using_starrocks/caching/block_cache.md • data_source/data_cache.md • knowledge/usage/best-practice.md
b
@Rocky Should I update the cache to be 100% of the disk too since right now its at 80%? I have a separate spill disk as well
r
Setting the data cache to 100% of the disk is generally not recommended, even if you have a separate disk for spilling. Here is why you should stick to a limit (typically 80-90%) and how to think about your specific setup: 1. Risk of "Disk Full" Errors Even on a dedicated cache disk, there are factors that require some "breathing room": * Filesystem Overhead: Every filesystem (XFS, EXT4) needs some free space for metadata and efficient block allocation. If a disk is 100% full, the filesystem can become extremely slow or even read-only. * Metadata Storage: StarRocks stores cache metadata (indexing where blocks are located) which also consumes space. * Logs and Other Files: If your 
storage_root_path
or log directories happen to point to the same disk partition, they will compete for space. If the cache takes 100%, the node might crash when it tries to write a log or a core dump. 2. The 80% Default and Eviction StarRocks' Data Cache is designed to be an LRU (Least Recently Used) cache. It doesn't actually delete files to free up space; it marks old data as overwritable once the limit is reached. * Default Behavior: By default, StarRocks targets 80% usage. * Observation: You might notice that even with an 80% limit, the actual OS-level disk usage (reported by 
df -h
) might look higher (e.g., 85-90%) because of the way the cache manages peak historical usage. 3. Impact of the Separate Spill Disk Having a separate spill disk is an excellent practice. It ensures that when a large query triggers a spill to disk, it doesn't: 1. Contend for I/O with the data cache. 2. Compete for space with the data cache. Since your spill disk is separate, you can safely increase the cache limit from 80% to 90% if you want to maximize your cache hit rate, but avoid 100%. Recommended Action If you want to squeeze more performance out of your dedicated cache disk, you can update the configuration to 90%. You can do this dynamically for the BE/CN nodes using:
Copy code
sql
-- Replace <BE_ID> with your actual Backend ID
UPDATE be_configs SET VALUE="90" WHERE NAME="starlet_star_cache_disk_size_percent" AND BE_ID=<BE_ID>;
Summary Checklist: * 80-90%: Safe and recommended. * 95%+: Risky; may lead to filesystem fragmentation or "No space left on device" errors for system tasks. * Separate Spill Disk: Keep this as is—it’s the best way to prevent spill-heavy queries from killing your cache performance. References • using_starrocks/caching/block_cache.md • administration/management/resource_management/spill_to_disk.md • data_source/data_cache_troubleshooting.md
2 Views
Open in Slack
PreviousNext
Powered by