http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> How do I set `enable_spill` to `tru...
# questions-and-troubleshooting
d
@Rocky How do I set 
enable_spill
to 
true
using a configuration file? I've created a config map, set 
enabled_spill = true
and told my CN node to load the config map, however, when I query the global properties table, I see it set to 
false
. Have I done something wrong, or does that table not show the the configurations of the CN nodes?
r
Setting 
enable_spill
to 
true
in a CN configuration file (like 
cn.conf
or a K8s ConfigMap for CN) will not work because 
enable_spill
is a System Variable, not a node configuration parameter. Here is how you can correctly enable and verify spilling: 1. Where to set the configurations In StarRocks, spilling requires two distinct configuration steps: * Node Level (CN/BE): You must define where to store the spilled data. This is done in the configuration file (
cn.conf
or your ConfigMap).
Copy code
properties
    # Add this to your CN ConfigMap/cn.conf
    spill_local_storage_dir=/path/to/spill/data
* Cluster/Session Level (FE): You must tell the query engine to use spilling. This is a system variable managed by the Front End (FE) and must be set via SQL.
Copy code
sql
    -- This persists globally across restarts
    SET GLOBAL enable_spill = true;
2. Why your current approach failed * Wrong Location: The CN node does not manage the 
enable_spill
variable; the FE does. When you set it in the CN ConfigMap, the CN ignores it, and the FE has no way of knowing you intended to change a global variable. * Typo: In your query, you mentioned setting 
enabled_spill
. The correct variable name is 
enable_spill
 (no 'd'). * Variable Scope: The "global properties table" (likely 
information_schema.global_variables
or 
SHOW GLOBAL VARIABLES
) reflects the state maintained by the FE. It does not look at individual CN/BE configuration files for these settings. 3. Recommended Workflow To correctly enable spilling on your CN nodes: 1. Update your CN ConfigMap to include the storage path (if not already set):
Copy code
properties
    spill_local_storage_dir=/opt/starrocks/cn/spill
2. Restart your CN nodes so they recognize the storage directory. 3. Connect to StarRocks via a MySQL client and run:
Copy code
sql
    SET GLOBAL enable_spill = true;
4. Verify the setting:
Copy code
sql
    SHOW GLOBAL VARIABLES LIKE 'enable_spill';
Note: If you want to test spilling immediately without changing the global state for all users, you can set it at the session level for your current connection: 
SET enable_spill = true;
. References • administration/management/FE_configuration.md
d
@Rocky What is the default 
spill_local_storage_dir
location?
r
By default, the 
spill_local_storage_dir
parameter does not have a pre-configured location in a standard StarRocks installation. You are generally required to specify it manually to ensure that spilling doesn't accidentally fill up your system partition. However, depending on how you deployed StarRocks, there are common "effective" defaults: 1. Kubernetes / Helm Deployment (Most Likely) If you are using the StarRocks Operator or Helm charts (which your mention of a "ConfigMap" suggests), the default path used by the container is: * 
/opt/starrocks/cn/spill
 (for CN nodes) * 
/opt/starrocks/be/spill
 (for BE nodes) In many K8s configurations, the Helm chart will automatically mount an 
emptyDir
or a Persistent Volume at this path and inject the configuration for you. 2. Standard Linux Deployment If you installed StarRocks manually on Linux, the parameter is empty by default. If you enable 
enable_spill = true
without defining this directory in 
cn.conf
, the spilling will fail when triggered because the node has no designated scratch space. 3. Relationship with 
query_scratch_dirs
 Before 
spill_local_storage_dir
was introduced, StarRocks used 
query_scratch_dirs
. * Default: 
${STARROCKS_HOME}
(the root directory of your StarRocks installation). * Note: Modern versions of StarRocks (v3.0+) prefer 
spill_local_storage_dir
. Recommendation If you are manually setting this in your ConfigMap, it is best practice to point it to a dedicated disk or a high-performance mount point to avoid I/O contention with your logs or data. Example ConfigMap entry:
Copy code
properties
spill_local_storage_dir=/data/starrocks/spill
Verification: To check if the CN has correctly picked up your directory setting, you can check the node's configuration via the 
proc
file system (replace 
<cn_host>
and 
<cn_http_port>
):
Copy code
sql
-- Run this in your SQL client
ADMIN SHOW FRONTEND CONFIG; -- For FE
-- To see BE/CN configs specifically:
SHOW BACKENDS; -- Look at the 'IP' and 'HttpPort'
-- Then visit http://<cn_host>:<cn_http_port>/var in a browser or use curl
``` ## References - [administration/management/BE_configuration.md](https://docs.starrocks.io/docs/administration/management/BE_configuration) - [administration/management/resource_management/spill_to_disk.md](https://docs.starrocks.io/docs/administration/management/resource_management/spill_to_disk) - [sql-reference/System_variable.md](https://docs.starrocks.io/docs/sql-reference/System_variable)
d
@Rocky How to determine if my starrocks cluster's performance is limited by FE nodes, CN nodes or my table's data?
r
To determine where your performance bottleneck lies, you should use the Query Profile. This is the single most important diagnostic tool in StarRocks. You can obtain it for a query by running:
Copy code
sql
-- For a finished query
SHOW PROFILELIST; -- Find the QueryId
ANALYZE PROFILE FROM '<QueryId>';

-- For the next query you run in this session
SET enable_profile = true;
-- Run your SQL
Here is how to interpret the results to find your bottleneck: 1. Is it the FE (Front End)? The FE handles query parsing, planning, and metadata management. * Profile Indicators: Look at the Query Planning Time in the summary section. If this is a significant portion of the total time (e.g., query takes 2s and planning takes 1.5s), the FE is the bottleneck. * Common Causes: * Complexity: Extremely large SQL with hundreds of joins. * Metadata Pressure: Your table has too many partitions or tablets (thousands+), making the planner work harder to find data locations. * FE Resources: Check the FE node's CPU/Memory. If Java GC (Garbage Collection) is high, planning slows down. 2. Is it the CN (Compute Node)? The CN/BE handles the actual execution logic (aggregations, joins, expressions). * Profile Indicators: Look at the Fragments and Operators. * CPU-Bound: If operator time (like 
AGGREGATE
or 
JOIN
) is high and CPU usage on the CN is near 100%. * Memory/Spilling: If you see 
PeakMemory
hitting the node limit or counters for 
SpillWriteRows
/ 
SpillReadRows
. This means the CN is struggling to keep data in memory. * Exchange Bottleneck: If 
EXCHANGE_NODE
(data transfer between nodes) takes a long time, your network might be saturated or your join distribution is inefficient. * Hardware Check: Run 
top
or check Grafana for the CN nodes. High CPU usage across all nodes suggests you need more CN power. 3. Is it the Table Data / Design? This is often the most common bottleneck and manifests as slow Scans. * Profile Indicators: Look at the 
OLAP_SCAN_NODE
 metrics. * Data Skew: Check the 
Max/Min/Avg
time across different CN nodes for the same fragment. If one node takes 10s and others take 1s, your data is skewed (some buckets are much larger than others). * Inefficient Filtering: Check 
RowsRead
vs 
RowsReturned
. If you read 1 billion rows but only return 100, you likely need better Sort Keys, Bloom Filters, or Bitmap Indexes. * I/O Bottleneck: High 
ScanTime
but low CPU on the CN node. This means the node is waiting for the disks. * Small Files: If 
OpenFileTime
or 
IOTime
is high, you might have too many small files (check your compaction status). * Design Check: Verify your 
DISTRIBUTED BY
key. If you use a key with low cardinality (like 
gender
), your data will not be distributed evenly across CN nodes. Summary Diagnostic Table Symptom | Primary Suspect | Tool to Check Long "Planning Time" | FE Node | 
ANALYZE PROFILE
(Summary) High CPU on all CNs | CN/BE Capacity | Linux 
top
/ Grafana High "ScanTime" | Table Design (Indices/Sort Keys) | 
ANALYZE PROFILE
(Scan Node) One CN node much slower | Data Skew (Bucket Key) | 
SHOW TABLE STATUS
/ Profile stats High "Exchange" Time | Network / Join Strategy | 
EXPLAIN
(Shuffle vs Broadcast) Rocky's Tip: Always check 
EXPLAIN ANALYZE <your_query>
 first. It provides a visualized breakdown of where the time was spent directly spent in the execution graph. References • best_practices/query_tuning/query_profile_overview.md • best_practices/query_tuning/query_profile_text_based_analysis.md • knowledge/usage/best-practice.md
d
@Rocky I can see the 
enable_spill
is true when I run 
SHOW GLOBAL VARIABLES LIKE 'enable_spill'
but the 
spill
directory of my cn nodes are empty, even though the nodes themselves keep crashing
r
If your 
spill
directory is empty despite 
enable_spill
being set to 
true
, it is likely due to one of the following three reasons: 1. Spilled data is automatically deleted StarRocks handles spilled data as temporary intermediate files. * The behavior: As soon as a query finishes (successfully or with an error), the CN/BE node automatically deletes all files in the spill directory associated with that query. * How to verify: You would need to check the directory while a heavy query is actively running. If you have a very large query, run it and use a command like 
watch -n 1 ls -R /path/to/spill
to see if temporary files appear and disappear. 2. The 
spill_mode
is set to 
auto
(Default) By default, even if 
enable_spill
is 
true
, StarRocks will only spill data to disk if the query actually hits a memory pressure threshold. * The behavior: If your queries are small enough to fit in the CN's memory, StarRocks will keep everything in RAM for performance and never write to the disk. * How to test: You can force StarRocks to use the disk regardless of memory usage by running:
Copy code
sql
    SET spill_mode = 'force';
    -- Run a query with a large SORT or JOIN after this
In 
force
mode, every supported operator (Sort, Aggregation, Join) will write to your spill directory immediately. 3. The query operators don't support spilling Not every SQL operation can spill to disk. Spilling is currently supported for: * Aggregate operators (e.g., 
GROUP BY
) * Sort operators (e.g., 
ORDER BY
) * Hash Join operators (Join operations) * CTE operators (From v3.3.4+) If your query is a simple 
SELECT *
or a light filter, it will never trigger a spill. ────────── How to confirm if spilling happened (even if files are gone) Instead of looking at the folder, check the Query Profile. This is the source of truth for whether the engine attempted to spill. 1. Run your query. 2. Get the profile:
Copy code
sql
    -- Get the ID of your last query
    SHOW PROFILELIST;
    -- Analyze the specific query
    ANALYZE PROFILE FROM 'your_query_id';
3. Look for these specific metrics in the output: * 
SpillWriteRows
: Number of rows written to disk. * 
SpillReadRows
: Number of rows read back from disk. * 
SpillWriteBytes
: Total data volume written to your directory. If these metrics are 
0
or missing, the query did not spill. If they are greater than 
0
, the directory was used and then cleaned up as expected. A Note on Permissions Ensure the user running the StarRocks process has read/write permissions for the 
spill_local_storage_dir
. If the directory is owned by 
root
and StarRocks is running as a 
starrocks
user, it will fail to write and usually log an "Access Denied" or "IO Error" in the 
<http://cn.INFO|cn.INFO>
or 
<http://be.INFO|be.INFO>
logs. References • administration/management/resource_management/spill_to_disk.md • sql-reference/System_variable.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md
If your CN nodes are crashing while memory-intensive queries are running, even with 
enable_spill = true
, it typically means the memory usage is growing faster than the spilling mechanism can react, or the memory is being consumed by non-spillable components. Here are the specific reasons why your spill directory is empty and the nodes are still crashing: 1. The "100% Threshold" Problem In many versions of StarRocks, the default threshold to trigger spilling is very high. * The Parameter: 
query_pool_spill_mem_limit_threshold
(in 
cn.conf
) defaults to 1.0 (100%). * The Crash: If the threshold is 1.0, StarRocks only starts spilling when the query pool is already full. By the time the system starts writing data to disk, the Linux kernel's OOM Killer or StarRocks' internal watchdog may have already killed the process to protect the system. * Fix: Lower the threshold in 
cn.conf
so spilling starts before the node is at risk:
Copy code
properties
    # Trigger spilling when 80% of the query pool is used
    query_pool_spill_mem_limit_threshold = 0.8
2. Automatic Cleanup on Restart StarRocks is designed to be self-cleaning. * The Behavior: If a CN node crashes, StarRocks assumes there might be "garbage" left in the spill directory. Upon restarting, the node automatically scans and deletes all files in the 
spill_local_storage_dir
to ensure it doesn't run out of disk space. * Why it looks empty: Even if the node managed to spill 10GB before crashing, the directory will be empty by the time you log back in to check it. 3. Non-Spillable Memory Usage Spilling only works for specific operators: Aggregates, Sorts, and Joins. * The Limitation: Memory used for Expression Evaluation (complex calculations in the 
SELECT
or 
WHERE
clause) or Metadata cannot be spilled. * The Crash: If your query is crashing because of a massive expression (e.g., a huge 
IN
list or complex string manipulation), 
enable_spill
will not help, and the directory will remain empty. 4. Spilling is Too Slow If your disk I/O (where the spill directory is located) is slow, the memory usage might continue to climb while the node is waiting to write data to the disk. * Diagnosis: Check if your 
spill_local_storage_dir
is on a network drive or a slow HDD. It should ideally be on local NVMe/SSD. ────────── How to Fix and Diagnose Step 1: Force a Spill to Test Configuration To verify your directory is actually writable and the configuration works, run a query with 
force
mode. This bypasses memory checks:
Copy code
sql
SET spill_mode = 'force';
-- Run a large query
SELECT count(*) FROM large_table GROUP BY large_column;
If the directory is still empty during this query, check your 
<http://cn.INFO|cn.INFO>
logs for "Permission Denied" or "No such directory" errors. Step 2: Check for OOM Kills Check your Linux system logs to see if the OS is killing the process:
Copy code
bash
sudo dmesg -T | grep -i oom
# OR
grep -i "killed process" /var/log/messages
Step 3: Check CN Logs for "Spill" Grep your CN logs for "spill" to see if the engine even tried to start the process before it died:
Copy code
bash
grep -i "spill" /path/to/starrocks/log/cn.INFO
Step 4: Adjust Memory Settings If the nodes are crashing, ensure your 
mem_limit
in 
cn.conf
leaves enough room for the OS. For example, if the machine has 64GB RAM, set 
mem_limit = 90%
. Rocky's Tip: If you are using Resource Groups, ensure you have set the 
spill_mem_limit_threshold
inside the resource group definition, as that overrides global settings. References • administration/management/resource_management/spill_to_disk.md • administration/management/BE_configuration.md
d
@Rocky what data do FE nodes cache?
r
In StarRocks, the FE (Front End) acts as the brain of the cluster. While it doesn't store the actual data rows (which live on BE/CN nodes), it maintains several critical caches in memory to speed up query planning, metadata access, and transaction management. Here is a breakdown of what the FE nodes cache: 1. Catalog & Table Metadata (The "Catalog Cache") The FE keeps the entire state of the cluster's metadata in memory. This is the most significant "cache" on the FE. * Internal Tables: Database schemas, table structures, partition ranges, and tablet locations (which BE/CN has which tablet). * User & Permissions: Roles, user accounts, and access control lists (ACLs) are cached for instant authentication. * External Catalogs (Hive, Iceberg, Hudi): For data lake queries, the FE caches file lists and schema information from external metastores to avoid expensive network calls for every query. 2. External Metadata Cache (Version 3.x+) For data lake integration (specifically Iceberg and Hive), the FE uses a specialized two-level cache: * Memory Cache: Caches deserialized metadata objects (like Iceberg manifest files or Avro objects) to skip parsing overhead. * Disk Cache: If enabled (
enable_iceberg_metadata_disk_cache
), the FE can spill these metadata files to its local disk to handle massive datasets that exceed RAM capacity. 3. Statistics Cache The Cost-Based Optimizer (CBO) relies on statistics (Min/Max, Null counts, NDV) to create efficient query plans. * What's cached: Data from the internal 
_statistics_
table. * Key Parameters: 
statistic_cache_columns
(controls how many column-stats are kept in memory). This cache is refreshed periodically via 
statistic_update_interval_sec
. 4. Query Cache Metadata (Coordination) While the actual results of a query are cached on the BE/CN nodes, the FE caches the "fingerprints" of queries. * Normalization: The FE normalizes SQL (removing whitespace/case differences) and checks if a query is eligible to reuse results stored on the BEs. 5. Stream Load Coordinator Cache For high-frequency data loading: * Transaction Mapping: Caches the mapping from a transaction label to the specific coordinator node. This prevents "label already exists" errors and helps manage concurrent transactions. * Key Parameter: 
transaction_stream_load_coordinator_cache_capacity
. 6. SQL Parse & Plan Cache To reduce CPU overhead during query planning: * Short-circuiting: For simple queries (like 
SELECT * FROM table WHERE id = 1
), the FE can cache certain planning phases to return results faster. ────────── What FE nodes do NOT cache: * Query Result Sets: StarRocks does not have a "Result Cache" on the FE. Final result sets are streamed from the BEs to the FE and then directly to the client. * Data Blocks: The "Block Cache" (which caches Parquet/ORC data from S3/HDFS) resides entirely on the BE or CN nodes, never the FE. Rocky's Tip: If your FE is using too much memory, it's usually due to having a massive number of partitions or tablets. You can check the FE memory usage and heap settings if the metadata for your external catalog is very large. References • administration/management/FE_configuration.md • data_source/catalog/iceberg/iceberg_catalog.md • using_starrocks/caching/query_cache.md • faq/Sql_faq.md • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md -management) management) management) References
d
@Rocky What data spills to disk for FE nodes when caching data? What is an appropriate directory to add a PVC to?
r
For FE (Front End) nodes, the concept of "spilling" and "caching to disk" is specific to metadata, as FE nodes do not handle the actual query data (that is done by BE or CN nodes). 1. What data spills/caches to disk on FE? There are two main types of data that the FE writes to disk: * Cluster Metadata (Critical): This is the "brain" of your cluster. It includes the schema, user permissions, tablet locations, and transaction logs (edit logs). This is stored in the 
meta_dir
. * External Metadata Cache (Iceberg): For Data Lake queries, specifically Iceberg, StarRocks uses a two-level cache. If the metadata (manifest files) is too large for memory, it "spills" to disk. * Trigger: Controlled by the configuration 
enable_iceberg_metadata_disk_cache = true
. * Content: Deserialized Iceberg metadata objects (Avro/JSON). * Materialized View (MV) Spill: Starting from v3.1.1, FE can spill intermediate results during Materialized View refresh tasks if 
enable_materialized_view_spill
is enabled. ────────── 2. Appropriate Directories for PVCs In a Kubernetes environment, you should attach PVCs to the following directories on the FE node. The default paths are relative to 
${STARROCKS_HOME}
(usually 
/opt/starrocks/fe
): Purpose | Default Path | PVC Recommendation Cluster Metadata | 
/meta
| Mandatory. This must be persistent. If this data is lost, your cluster is lost. Metadata Cache | 
/caches/iceberg
| Optional. Recommended if you use Iceberg catalogs heavily and have enabled disk caching. Logs | 
/log
| Recommended. Useful for troubleshooting crashes and long-term auditing. Small Files | 
/small_files
| Optional. Stores files uploaded via the 
SUBMIT
command or certificates. Summary Recommendation for PVC Configuration If you are using the StarRocks Helm Chart or Operator, the standard practice is to define a PVC for the meta directory at a minimum. Example mount points in a Pod spec:
Copy code
yaml
volumeMounts:
  - name: fe-meta
    mountPath: /opt/starrocks/fe/meta
  - name: fe-log
    mountPath: /opt/starrocks/fe/log
  # If you use Iceberg Disk Caching:
  - name: fe-cache
    mountPath: /opt/starrocks/fe/caches
Important Note: Do not confuse the FE Metadata Cache with the BE Data Cache. The BE/CN nodes use a "Block Cache" (storing actual data from S3/HDFS) which requires much larger PVCs and is configured via 
storage_root_path
. The FE cache is purely for the "map" of where that data lives. References • administration/management/FE_configuration.md • data_source/catalog/iceberg/iceberg_catalog.md • quick_start/helm.md • administration/management/resource_management/spill_to_disk.md • administration/stargo.md References • administration/management/FE_configuration.md • quick_start/helm.md • administration/stargo.md
d
@Rocky What is the default and recommended value for data cache disk size? I read the default is 80% but when i checked my nodes i saw 100%, for example:
Copy code
select * from information_schema.be_configs WHERE NAME="datacache_disk_size";
+-------+---------------------+-------+--------+---------+---------+
| BE_ID | NAME                | VALUE | TYPE   | DEFAULT | MUTABLE |
+-------+---------------------+-------+--------+---------+---------+
| 10005 | datacache_disk_size | 100%  | string | 100%    |       1 |
| 12144 | datacache_disk_size | 100%  | string | 100%    |       1 |
| 10006 | datacache_disk_size | 100%  | string | 100%    |       1 |
| 12143 | datacache_disk_size | 100%  | string | 100%    |       1 |
+-------+---------------------+-------+--------+---------+---------+
r
The discrepancy you're seeing between the documentation (80%) and your cluster (100%) is usually due to the Automatic Scaling feature introduced in recent versions (v3.3+). 1. Recommended Value: 80% The recommended value is 80%. This is considered the "Safe Level." Setting it to 80% ensures that 20% of your disk remains free for: * System Logs: 
<http://cn.INFO|cn.INFO>
, 
<http://be.INFO|be.INFO>
, and audit logs. * Temporary Files: Spill-to-disk files (if enabled). * Metadata: Primary Key indexes and tablet metadata. * System Stability: Prevents the OS from slowing down or crashing due to a completely full disk. 2. Why you see "100%" as Default In StarRocks v3.3 and later, if you do not explicitly set 
datacache_disk_size
in your 
cn.conf
or 
be.conf
, the system enables Automatic Scaling (
datacache_auto_adjust_enable = true
) by default. When auto-scaling is active, the 
datacache_disk_size
variable often reports 
100%
because it tells the system: "You have permission to manage up to the full capacity, but follow the internal safety thresholds." Even if it shows 100%, the internal Automatic Scaling logic uses these default safeguards: * 
datacache_disk_high_level
(Default 90%): The node will start aggressively evicting old cache data once the disk reaches 90% usage. * 
datacache_disk_safe_level
(Default 80%): The system aims to keep the cache at this level after cleaning up. * 
datacache_disk_low_level
(Default 60%): The system will expand the cache if it is full and disk usage is below 60%. 3. How to check your actual effective limit To see if your node is actually respecting the 80/90 rule despite the "100%" setting, check these related configurations:
Copy code
sql
SELECT * FROM information_schema.be_configs 
WHERE NAME IN ('datacache_auto_adjust_enable', 'datacache_disk_high_level', 'datacache_disk_safe_level');
4. Rocky's Advice If you want to be safe and ensure the disk never fills up (especially since you mentioned nodes crashing earlier), I recommend manually overriding it to 80%. This turns off the "elastic" expansion and provides a hard cap. To change it dynamically (won't persist after restart):
Copy code
sql
UPDATE be_configs SET VALUE="80%" WHERE NAME="datacache_disk_size";
To make it permanent: Add this to your 
cn.conf
or `be.conf`:
Copy code
properties
datacache_disk_size = 80%
References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • knowledge/usage/best-practice.md References • using_starrocks/caching/block_cache.md • data_source/data_cache.md • data_source/data_cache_troubleshooting.md
d
@Rocky I don't see those properties in 4.1
r
In StarRocks v4.0 and later, several Data Cache configuration parameters were renamed to be more concise. If you are on version 4.1, you won't find the old 
datacache_...
names for the auto-scaling thresholds. 1. The Rename Map (v4.0+) Here is why you don't see the previous properties: Old Name (v3.3) | ***New Name (v4.0+)*** | Default Value 
datacache_auto_adjust_enable
| 
enable_datacache_disk_auto_adjust
 | 
true
datacache_disk_high_level
| 
disk_high_level
 | 
90
(90%) 
datacache_disk_safe_level
| 
disk_safe_level
 | 
80
(80%) 
datacache_disk_low_level
| 
disk_low_level
 | 
60
(60%) 2. Why 
datacache_disk_size
is still "100%" The 
datacache_disk_size
parameter itself was not renamed, but its default remains 
100%
. When 
enable_datacache_disk_auto_adjust
is set to 
true
(which is the default), the "100%" value for 
datacache_disk_size
essentially acts as an infinite quota, telling the system: _"You are allowed to use the whole disk if needed, but please manage the actual usage based on the 
disk_high_level
and 
disk_safe_level
thresholds."_ 3. How to check them in v4.1 To see your current effective settings in version 4.1, run this query:
Copy code
sql
SELECT * FROM information_schema.be_configs 
WHERE NAME IN (
    'datacache_disk_size', 
    'enable_datacache_disk_auto_adjust', 
    'disk_high_level', 
    'disk_safe_level', 
    'disk_low_level'
);
Rocky’s Recommendation for v4.1 Even though the default 
disk_safe_level
is 80, if you want to ensure the cache never tries to claim more than a specific amount of space regardless of auto-scaling logic, you can still hard-set the size:
Copy code
sql
-- Hard limit the cache to 80% of the disk capacity
UPDATE be_configs SET VALUE="80%" WHERE NAME="datacache_disk_size";
This prevents the "elastic" behavior where the cache might briefly spike to 90% (
disk_high_level
) before the eviction process catches up. References • administration/management/BE_configuration.md • data_source/data_cache.md • release_notes/release_4.0.md (Note: Assuming 4.0 naming conventions persist in the user query refers to the v4.0 series) References • data_source/data_cache.md • administration/management/BE_configuration.md
d
@Rocky With automated snapshots turned on, does StarRocks delete old snapshots automatically. Or do I need a expiration policy for my s3 bucket?
r
The answer depends on which feature you are using, as StarRocks has two distinct snapshot/backup mechanisms: 1. Automated Cluster Snapshot (v3.4+) If you are using the feature enabled via 
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON
(typically for shared-data clusters), StarRocks manages the deletion automatically. * Behavior: The system maintains only the most recent state. Each time a new snapshot is successfully created (based on the 
automated_cluster_snapshot_interval_seconds
setting), the previous historical snapshot is automatically dropped by the FE. * Storage: It keeps only one valid snapshot at any given time. * Bucket Policy: You generally do not need an S3 expiration policy for this feature because StarRocks handles the lifecycle of these specific files. 2. Standard 
BACKUP
Snapshots If you are running 
BACKUP SNAPSHOT ...
commands (manually or via a cron job/script), StarRocks does NOT delete old snapshots automatically. * Behavior: Each 
BACKUP
command creates a new, independent recovery point in your S3 repository. StarRocks does not track "expiration" for these backups. * Documentation Warning: The official documentation explicitly states: "Data snapshots backed up in the remote storage system cannot be deleted through StarRocks. You need to delete them manually in the remote storage system." * Recommendation: * S3 Lifecycle Policy: This is highly recommended to prevent storage costs from ballooning. You can set a policy to delete objects in the backup prefix after $X$ days. * Manual Cleanup: You can see what is in your repository with 
SHOW SNAPSHOT ON <repo_name>
, but you must delete the physical files via the S3 console/CLI. Summary Table Feature | Automated Deletion? | Cleanup Method Automated Cluster Snapshot | Yes | Automatic (Only keeps the latest). Standard 
BACKUP
Command | No | S3 Lifecycle Policy (Recommended) or manual deletion. Note on BE "Local" Snapshots You might see a configuration called 
snapshot_expire_time_sec
(default 48 hours) in your BE settings. This only affects temporary snapshot files stored on the local BE disks during the backup process or manual tablet cloning. It does not affect files already uploaded to S3. References • administration/cluster_snapshot.md • administration/management/Backup_and_restore.md • faq/Exporting_faq.md References • faq/Exporting_faq.md • administration/management/Backup_and_restore.md • administration/cluster_snapshot.md
2 Views
Open in Slack
PreviousNext
Powered by