http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I need to urgently cleanup space o...
# questions-and-troubleshooting
a
@Rocky I need to urgently cleanup space on drives holding metadata, logs and temp files for CN nodes. Shared-data mode 3.5.15. What options do I have?
r
For StarRocks version 3.5.15 in shared-data mode, CN (Compute Nodes) are designed to be stateless, but they can accumulate significant data on local disks due to logs, temp (spill) files, and data cache (metadata and blocks). Here are the options to urgently clean up space on your CN drives: 1. Cleanup Logs Logs are typically stored in 
${STARROCKS_HOME}/log
. * Manual Cleanup: You can safely delete older rotated log files (e.g., 
cn.INFO.log.2023...
) using 
rm
. * Configuration (cn.conf): * `sys_log_roll_num`: Reduce this (default is 10) to keep fewer historical log files. * `load_error_log_reserve_hours`: Reduce this (default is 48) to clean up import error logs faster. These are stored in 
storage/error_log
. * Audit Logs: If you have enabled audit logging to local files, check the 
audit
directory and prune old files. 2. Cleanup Data Cache (Metadata & Blocks) In shared-data mode, CNs cache remote data locally in the path defined by 
storage_root_path
(default is 
${STARROCKS_HOME}/storage
). This includes the 
starlet_cache
(data blocks) and 
starlet_meta
(cache metadata). * Option A: Dynamic Cleanup (No Restart) You can trigger an automatic cleanup by temporarily setting the cache quota to 0 using SQL. This forces the CN to evicted all cached data and metadata.
Copy code
sql
    -- Replace <CN_ID> with the ID from 'SHOW COMPUTE NODES'
    UPDATE be_configs SET VALUE = "0" WHERE NAME = "datacache_disk_size" AND BE_ID = <CN_ID>;

    -- After disk space is freed, set it back to your desired size (e.g., 1TB)
    UPDATE be_configs SET VALUE = "1024G" WHERE NAME = "datacache_disk_size" AND BE_ID = <CN_ID>;
* Option B: Manual Cleanup (Restart Required) 1. Stop the CN. 2. 
rm -rf <storage_root_path>/starlet_cache/*
3. 
rm -rf <storage_root_path>/starlet_meta/*
4. Start the CN. 3. Cleanup Temp Files (Spill to Disk) If you have queries that spill to disk (intermediate results), they use the directory defined by 
spill_local_storage_dir
(or 
query_scratch_dirs
in older configs). * Automatic: StarRocks usually cleans these up after a query finishes or upon restart. * Manual: If the CN crashed or the cleanup failed, check your spill directory (default often inside 
${STARROCKS_HOME}
) and look for directories named 
spill
. You can safely delete files in these directories only if no queries are currently running. 4. Empty the Trash Folder Deleted data files or temporary files are often moved to a 
trash
folder before final deletion. * Manual: Delete the contents of 
<storage_root_path>/trash/
. * Configuration (cn.conf): * `trash_file_expire_time_sec`: Reduce this value (default is 86400, or 24 hours) to make the system auto-clean the trash more frequently. Summary Checklist for Urgent Action: 1. Run 
rm -rf ${STARROCKS_HOME}/log/_.log.202_
 to clear old system logs. 2. *Run `rm -rf ${STARROCKS_HOME}/storage/trash/*`* to clear the trash. 3. Use the SQL 
UPDATE be_configs
 method to wipe the data cache without stopping the service. 4. Check for pprof artifacts in 
${STARROCKS_HOME}/log
(e.g., heap snapshots) and delete them if present. References • administration/management/BE_configuration.md • administration/management/resource_management/spill_to_disk.md • sql-reference/System_variable.md
a
@Rocky - starrocks, CN node - UPDATE be_configs SET VALUE = "0" WHERE NAME = "datacache_disk_size" AND BE_ID = <CN_ID>; command does not go through as "table be_confis is not found"
r
The error "table be_configs is not found" occurs because you need to specify the 
information_schema
 database prefix. In StarRocks, system configurations for Backend (BE) and Compute (CN) nodes are managed through a virtual table in that schema. To urgently clean up space on your CN nodes (version 3.5.15), use the following methods: 1. Dynamic Cleanup of Data Cache (No Restart) This is the most effective way to reclaim space used by cached data blocks from remote storage. Scaling the quota to 
0
forces the system to evict the cache. Corrected SQL Command:
Copy code
sql
-- You must include 'information_schema.'
UPDATE information_schema.be_configs 
SET VALUE = "0" 
WHERE NAME = "datacache_disk_size" AND BE_ID = <CN_ID>;

-- After the disk space is freed, set it back to your desired size (e.g., 500GB)
UPDATE information_schema.be_configs 
SET VALUE = "500G" 
WHERE NAME = "datacache_disk_size" AND BE_ID = <CN_ID>;
_Note: This change is not persistent. To make it permanent, update 
datacache_disk_size
in your 
cn.conf
file._ ────────── 2. Manual Cleanup of Logs Logs are the most common source of "hidden" disk pressure. * Path: 
${STARROCKS_HOME}/log
* Action: You can safely delete older rotated log files (e.g., 
cn.INFO.log.2024...
). * Command:
Copy code
bash
    # Delete all info logs older than today
    find /path/to/starrocks/log -name "cn.INFO.log.*" -type f -delete
* Prevention: In 
cn.conf
, set 
sys_log_roll_num = 5
to keep fewer historical files. ────────── 3. Cleanup Temp (Spill) Files If queries are spilling to disk due to memory pressure, they generate files in the directory specified by 
spill_local_storage_dir
(default is within 
storage_root_path
). * Manual Action: StarRocks typically cleans these up when a query ends. If the node crashed, "zombie" spill files might remain. * Safety Check: Only delete these if no large queries are currently running. Look for folders named 
spill
or 
query_scratch
inside your storage path and clear their contents. ────────── 4. Cleanup the Trash Folder When data is marked for deletion or internal moves, it often sits in a 
trash
folder for a safety period (default 24 hours). * Action: Manually delete the contents of the trash folder to reclaim space immediately. * Command:
Copy code
bash
    rm -rf <storage_root_path>/trash/*
* Prevention: In 
cn.conf
, reduce 
trash_file_expire_time_sec
(e.g., to 
3600
for 1 hour). ────────── 5. Clear Pprof Artifacts If you have performed profiling recently, large heap snapshots might be stored in the log directory. * Action: Check for and delete 
.pprof
or 
.heap
files in 
${STARROCKS_HOME}/log
. Summary Checklist 1. Run the SQL 
UPDATE information_schema.be_configs
 to wipe the Data Cache. 2. Delete rotated logs in the 
log/
directory. 3. Clear the 
trash/
folder in your 
storage_root_path
. 4. Check for pprof files if you've been debugging recently. References • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md • administration/management/BE_configuration.md • faq/Deploy_faq.md
a
@Rocky -
Copy code
UPDATE information_schema.be_configs 
SET VALUE = "0" 
WHERE NAME = "datacache_disk_size" AND BE_ID = <CN_ID>;
worked like a charm. Here is the question though. My datacache_disk_size was set to 200G. The underlying drive was 2TB. The CN cache took 1.9TB of space on that drive.
r
The reason your CN node used 1.9 TB despite setting 
datacache_disk_size
to 200 GB is due to how StarRocks calculates its cache quota and manages disk space. Here is why this happened and how to fix it: 1. The "Greater Of" Rule In StarRocks shared-data mode, the cache size is determined by the greater of two configuration parameters: * `datacache_disk_size`: Your hard-coded value (200 GB). * `starlet_star_cache_disk_size_percent`: A percentage of the total disk capacity. The default for 
starlet_star_cache_disk_size_percent
is 80%. On a 2 TB drive, 80% equals 1.6 TB. Since 1.6 TB is greater than your 200 GB setting, the system automatically used 1.6 TB as the effective cache limit, effectively ignoring your 200 GB cap. 2. Why it reached 1.9 TB (instead of 1.6 TB) While the cache target was likely 1.6 TB, the disk usage reached 1.9 TB because: * Peak Usage Persistence: Data Cache eviction does not actually delete files from the disk to reclaim space; it merely marks old blocks as "overwritable." The 
du
or 
df
commands will always show the "high-water mark" of historical usage. * Non-Cache Data: The remaining ~300 GB was likely occupied by: * Logs: System and audit logs in 
${STARROCKS_HOME}/log
. * Trash: Deleted files held in 
${storage_root_path}/trash
for 24 hours (default). * Spill files: Temporary data from large queries that "spilled" to disk. * Metadata: Cache metadata (
starlet_meta
) can grow significantly if you have millions of small cached blocks. How to enforce the 200 GB limit To ensure the CN strictly respects your 200 GB limit and doesn't default back to 80% of the disk, you must adjust your 
cn.conf
to disable the percentage-based calculation: 1. Update `cn.conf`:
Copy code
properties
    # Set the fixed size
    datacache_disk_size = 214748364800

    # Set percent to 0 so the fixed size is always "greater"
    starlet_star_cache_disk_size_percent = 0
2. Restart the CN node. Recommendations for Maintenance: * Monitor Logs: Check the size of your 
log
directory. If it's massive, set 
sys_log_roll_num = 5
in 
cn.conf
. * Lower Trash Expiry: Set 
trash_file_expire_time_sec = 3600
(1 hour) to clear the trash folder more aggressively. * Check Spill Dir: Ensure 
spill_local_storage_dir
is pointing to a drive with enough overhead for temporary query data. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md
a
@Rocky - would datacache_disk_size = 200G be accepted or do I need to go datacache_disk_size = 214748364800
r
Yes, 
datacache_disk_size = 200G
 is fully accepted and is the recommended way to set it. StarRocks' configuration parser understands human-readable units for this specific parameter. You do not need to calculate the exact number of bytes. Supported Formats for `datacache_disk_size`: * With Units: 
200G
, 
1T
, 
500GB
* Percentage: 
10%
(though in shared-data mode, this competes with 
starlet_star_cache_disk_size_percent
) * Bytes: 
214748364800
(also valid, but harder to read) Important Reminder To ensure this limit is strictly enforced and doesn't default back to 80% of your 2TB drive (which is 1.6TB), make sure you have both settings in your `cn.conf`:
Copy code
properties
# Strictly limit the cache to 200GB
datacache_disk_size = 200G

# Disable the percentage-based calculation so it doesn't override the 200G
starlet_star_cache_disk_size_percent = 0
Why this matters: StarRocks picks the greater of the two values. If you leave the percent at the default 80, it will see "200G vs 1.6TB" and choose 1.6TB every time the next time the node restarts. References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md • using_starrocks/caching/block_cache.md • data_source/data_cache.md
Open in Slack
PreviousNext
Powered by