http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> if cn pod using 50gb data cache pv...
# questions-and-troubleshooting
r
@Rocky if cn pod using 50gb data cache pvc, what disk spill size suitable for cn pod conf? and cache storage size?
r
For a StarRocks CN (Compute Node) pod with a 50GB PVC intended for both data cache and disk spilling, you need to carefully partition the space between these two features since they typically share the same local storage. Recommended Configuration If you have 50GB available, the following allocation is generally suitable: Purpose | Recommended Size | Configuration Parameter Data Cache | 30 GB - 35 GB | 
datacache_disk_size = 32212254720
(30GB in bytes) Disk Spill | Remaining (~15 GB) | No explicit limit parameter; uses remaining disk space ────────── Key Configuration Parameters (
cn.conf
) 1. Data Cache Size (
datacache_disk_size
): Limit the cache so it does not consume the entire PVC. If the cache fills the disk, your queries will fail when they need to spill intermediate results to disk.
Copy code
properties
    # Set to 30GB (32,212,254,720 bytes)
    datacache_disk_size = 32212254720
2. Spill Directory (
spill_local_storage_dir
): Ensure the spill directory is on the same PVC mount point as your data cache (typically under 
storage_root_path
).
Copy code
properties
    # Path where intermediate query results will spill
    spill_local_storage_dir = /opt/starrocks/cn/storage/spill
3. Storage Safety Limit (
storage_flood_stage_left_capacity_bytes
): Crucial: The default value for this is 100GB. Since your total disk is only 50GB, the node may refuse to write or report "disk full" immediately if you don't lower this.
Copy code
properties
    # Set to 2-5GB to ensure the CN keeps working on a small PVC
    storage_flood_stage_left_capacity_bytes = 2147483648
Why this allocation? * Stability over Performance: Disk spilling is a "correctness" feature that prevents queries from failing with Out-of-Memory (OOM) errors. Data cache is a "performance" feature. In a resource-constrained environment (50GB), it is better to have a slightly smaller cache than to have large queries crash because they ran out of spill space. * No Explicit Spill Limit: StarRocks currently does not have a dedicated 
spill_storage_limit
parameter for local disks. It simply uses the available space in the 
spill_local_storage_dir
. By limiting the 
datacache_disk_size
, you effectively "reserve" the remaining PVC space for spilling. Summary Checklist * PVC Mount: Ensure your PVC is mounted at the path specified by 
storage_root_path
(default is 
${STARROCKS_HOME}/storage
). * Enable Spill: Remember to enable the feature globally or per session:
Copy code
sql
    SET GLOBAL enable_spill = true;
References • administration/management/resource_management/spill_to_disk.md • administration/management/BE_configuration.md • sql-reference/System_variable.md
For a CN (Compute Node) pod with a 50GB PVC, you must carefully partition the space between the Data Cache and Disk Spill because they share the same local storage. By default, StarRocks might fail to write to a disk this small due to safety thresholds designed for larger production disks. Here is the recommended configuration for a 50GB volume: 1. Recommended Space Allocation With 50GB total, a balanced approach is to allocate 60–70% for Data Cache and leave the rest for Disk Spill and system overhead. Component | Recommended Size | Role Data Cache | 30 GB | Speeds up queries by caching remote data (S3/HDFS). Disk Spill | ~15–18 GB | Prevents OOM by spilling intermediate results of large queries. Reserved/Safety | 2–5 GB | Buffer to prevent "Disk Full" errors and node hang. ────────── 2. Critical Configuration (
cn.conf
) You must add or update these parameters in your 
cn.conf
. A. Fix the Disk Safety Threshold (Crucial) The default value for 
storage_flood_stage_left_capacity_bytes
is 100GB. Since your PVC is only 50GB, the CN will immediately think the disk is full and refuse to spill or cache data unless you lower this.
Copy code
properties
# Set safety threshold to 2GB (default is 100GB)
storage_flood_stage_left_capacity_bytes = 2147483648
B. Set Data Cache Size Explicitly limit the cache so it doesn't consume the entire PVC.
Copy code
properties
# Set cache limit to 30GB (value in bytes)
datacache_disk_size = 32212254720
C. Configure Spill Path Ensure the spill directory is pointing to your PVC mount point (usually under 
storage_root_path
).
Copy code
properties
# Path for intermediate query results
spill_local_storage_dir = /opt/starrocks/storage/spill
────────── 3. Summary of Parameters for 50GB PVC Parameter | Recommended Value | Description 
datacache_disk_size
| 
32212254720
(30GB) | The hard limit for the data cache. 
storage_flood_stage_left_capacity_bytes
| 
2147483648
(2GB) | Allows node to operate on a small disk. 
spill_local_storage_dir
| 
/path/to/pvc/spill
| Where spill files are stored. Why this allocation? * Preventing OOM: If you give too much space to the cache (e.g., 45GB), a single large query needing to spill only 5GB of intermediate data will fail with a "Disk Full" error. * Cache vs. Stability: The Data Cache is a performance optimization, while Disk Spilling is a stability feature. In small environments, it is safer to prioritize enough "breathing room" for spilling to ensure queries actually finish. * Sharing: StarRocks does not currently have a dedicated 
spill_disk_limit
parameter. By setting a hard limit on 
datacache_disk_size
, you effectively reserve the remaining disk space for the spill feature. Note: Ensure you enable spilling in your session or globally:
Copy code
sql
SET GLOBAL enable_spill = true;
References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/resource_management/Memory_management.md • deployment/plan_cluster.md
2 Views
Open in Slack
PreviousNext
Powered by