http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> several tables are stuck with high ...
# questions-and-troubleshooting
t
@Rocky several tables are stuck with high compaction scores and performing very poorly.
r
High compaction scores (specifically the MaxCS metric) indicate that StarRocks is unable to merge small data files (rowsets) as fast as they are being created. This typically results from frequent, small data loads and leads to significant query performance degradation. 1. Diagnose the Status First, identify which tables and partitions are most affected: * Check Compaction Scores:
Copy code
sql
    SHOW PARTITIONS FROM <table_name>;
    -- Or for all tables:
    SELECT * FROM information_schema.partitions_meta ORDER BY MAX_CS DESC LIMIT 10;
* MaxCS < 10: Healthy. * MaxCS > 100: Unhealthy; the system begins to slow down data loading. * MaxCS > 500: Critical; manual intervention is usually required. * Check Running/Stuck Tasks:
Copy code
sql
    SHOW PROC '/compactions';
    -- For shared-data (Cloud Native) clusters:
    SELECT * FROM information_schema.be_cloud_native_compactions;
If 
CommitTime
or 
FinishTime
is NULL for a long period, the task might be stuck. 2. Immediate Tuning (BE/CN Configuration) To help the cluster "catch up," you can increase the resources allocated to compaction. These can be modified dynamically: * Increase Compaction Threads: Set 
compact_threads
to approximately 25% of your CPU cores.
Copy code
sql
    UPDATE information_schema.be_configs SET value = <new_value> WHERE name = "compact_threads";
* Make Tasks Smaller & Faster: Lowering the number of files merged per task helps them finish faster and prevents OOM.
Copy code
sql
    UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";
3. Procedure for "Stuck" Compaction If compaction is completely stalled and scores aren't moving despite tuning, follow this recovery process: 1. Disable Compaction:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
2. Restart BE/CN nodes: This clears stale compaction tasks from memory. 3. Re-enable Compaction:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1"); -- -1 is auto-scaling
4. Root Cause Prevention High scores are almost always caused by small-file problems. To prevent recurrence: * Increase Load Intervals: Avoid loading data more frequently than every 10 seconds. * Increase Batch Size: Aim for larger row counts per load transaction. * Check Table Schema: For Primary Key tables, ensure your bucket count isn't too high, as this splits data into too many small tablets. If the problem persists, check the Leader FE log for "Publish failure" messages or 
TxnID
errors which might indicate why the final step of compaction is failing. References • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/BE_configuration.md
t
@Rocky several compaction jobs have been stuck for over an hour
r
If compaction jobs are stuck for over an hour, they are likely in a state where the compute nodes (BE/CN) have stopped reporting progress, or the final "Publish" phase is failing at the FE level. 1. Identify where they are stuck Run the following command to see the high-level status of all active compaction tasks:
Copy code
sql
SHOW PROC '/compactions';
* If 
CommitTime
is NULL: The compute nodes (BE/CN) are still trying to merge files or have crashed/stalled. * If 
FinishTime
is NULL but 
CommitTime
is set: The merge is finished, but the FE is failing to "Publish" the new version. Check the Leader FE log (
fe.log
) and search for the 
TxnID
from the command above to find the specific error. 2. Inspect Detailed Progress To see why a specific task isn't moving, check the tablet-level progress using the 
TxnID
from the previous step:
Copy code
sql
SELECT * FROM information_schema.be_cloud_native_compactions 
WHERE TXN_ID = <your_txn_id>;
* Check the 
PROGRESS
and 
STATUS
columns. * If 
PROGRESS
is not increasing, or if 
STATUS
shows an error (like "Timeout" or "S3 error"), you have found the bottleneck. 3. How to Clear Stuck Jobs Method A: Cancel the Transaction You can try to manually stop the specific stuck task from the Leader FE:
Copy code
sql
CANCEL COMPACTION WHERE TXN_ID = <your_txn_id>;
Method B: Clean Up Stale Tasks (Recommended if many are stuck) If multiple tasks are stuck and 
CANCEL COMPACTION
doesn't work, follow this "reset" procedure to clear the memory on the compute nodes: 1. Disable new compaction tasks:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
2. Restart the BE/CN nodes: This is necessary because FE does not always have full control over tasks already running on the backends. 3. Verify status: Run 
SHOW PROC '/compactions';
to ensure the list is now empty or tasks have failed. 4. Re-enable compaction:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1"); -- -1 for auto-scaling
4. Common Root Causes * Memory Issues: If the BE/CN nodes are low on memory, compaction threads may hang or be killed by the OOM killer. * Object Storage Latency: If using shared-data mode, high latency in S3/OSS can cause tasks to time out during the "Write" phase. * Small File Overload: If the "MaxCS" score is extremely high (e.g., >1000), the system may be overwhelmed. Consider increasing 
compact_threads
in the BE configuration once you have cleared the stuck jobs. References • knowledge/admin_and_operation/shared_data_compaction.md • faq/shared_data_faq.md • administration/management/compaction.md References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • faq/shared_data_faq.md
j
which version？
t
@Jeff Ding 4.1.4-4a9848e
Compactions seem stuck for several of these tables, I've turned the routine load interval up to 1min instead of 10 seconds to hopefully prevent this compaction. I haven't looked at all the tables, but the few I've checked pretty much all of the issues are on the latest partition (partitioned by month).
@Rocky lots of compactions queued, but CPU seems fairly idle
r
If your compaction tasks are queued while CPU usage is low, the system is being throttled by its internal concurrency limits. You need to increase the number of allowed parallel tasks and worker threads. 1. For Shared-Data (Cloud Native) Clusters In shared-data mode, the FE controls the global task limit, and the CN (Compute Node) controls the local execution threads. * Increase Global Concurrency (FE): By default, this is 
-1
(16 * number of CN nodes). If you have many tables/partitions, increase this to allow more tasks to be dispatched.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "128"); -- Adjust based on cluster size
* Increase Execution Threads (CN): The default is often only 4 threads. Recommendation is 25% of your CPU cores.
Copy code
sql
    -- Run this to update all nodes
    UPDATE information_schema.be_configs SET value = "16" WHERE name = "compact_threads";
2. For Shared-Nothing Clusters In traditional deployments, compaction is controlled per-node and per-disk. * Increase Disk-level Concurrency: If you have high-performance disks (SSDs/NVMe), the default of 1 thread per disk is usually too low.
Copy code
sql
    -- These require a BE restart (not dynamic in older versions)
    -- In be.conf:
    cumulative_compaction_num_threads_per_disk = 4
    base_compaction_num_threads_per_disk = 2
* Increase Global BE Threads:
Copy code
sql
    UPDATE information_schema.be_configs SET value = "16" WHERE name = "compact_threads";
3. Make Tasks Finish Faster Large compaction tasks can "hog" a thread for a long time. Reducing the task size makes them finish faster and rotate through the queue more quickly.
Copy code
sql
-- Reduces the number of files merged per task, preventing "heavy" slow tasks
UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";
4. Verify the Change After applying these, monitor the task throughput: * Check queue size: 
SHOW PROC '/compactions';
(Look at the number of tasks in 
RUNNING
vs 
PENDING
). * Check CPU utilization: It should now rise as more threads begin processing the queue. If CPU still remains idle after these changes, check if your BE nodes are hitting Memory Limits. Compaction will throttle itself if the BE's 
mem_limit
is nearly reached to prevent OOM crashes. References • administration/management/BE_configuration.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md
t
@Rocky it's a small cluster 2 CPU and 8GB of RAM across 3 compute nodes
r
With only 2 CPUs and 8GB of RAM per node, your cluster is at the absolute minimum resource level for StarRocks. In this environment, the "idle CPU while queued" behavior usually means compaction is being throttled by Memory Limits, not CPU availability. When StarRocks detects that a compaction task might exceed the available memory (which is very likely with only 8GB), it will pause or refuse to start new tasks to prevent the node from crashing (OOM). To get compaction moving on such small nodes, you must reduce the "size" of each individual task so they fit into your limited RAM: 1. Reduce Task Size (Critical) The default settings try to merge up to 1000 files at once. On an 8GB node, this will easily hit memory limits and stall.
Copy code
sql
-- Reduce the number of files merged per task to 50 or 100
UPDATE information_schema.be_configs SET value = "50" WHERE name = "max_cumulative_compaction_num_singleton_deltas";

-- For Primary Key tables specifically (if applicable)
UPDATE information_schema.be_configs SET value = "50" WHERE name = "lake_pk_compaction_max_input_rowsets";
2. Set Appropriate Thread Counts For a 2-core machine, having too many threads will cause context switching and memory contention.
Copy code
sql
-- Set threads to 1 or 2 (Maximum)
UPDATE information_schema.be_configs SET value = "1" WHERE name = "compact_threads";
3. Adjust Global Concurrency (Shared-Data) If you are using the Shared-Data (Cloud Native) version, the FE may be trying to schedule too many tasks for your small CN nodes to handle.
Copy code
sql
-- Limit the total number of concurrent tasks across the whole cluster
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "3");
4. Check for Memory Backpressure If tasks are still not moving, check if the BE is under memory pressure: 1. Run 
SHOW BACKEND\G
and look at the 
MemUsed
column. 2. If 
MemUsed
is near 6GB-7GB (out of 8GB), StarRocks will prioritize queries and stop compaction entirely. 3. Action: If memory is full, you may need to reduce your 
load_process_max_memory_limit_percent
(default 30%) to leave more room for compaction, or restart the BE nodes to clear the cache. Summary for Small Clusters Parameter | Recommendation (2 vCPU / 8GB) 
compact_threads
| 1 
max_cumulative_compaction_num_singleton_deltas
| 50 
lake_compaction_max_tasks
| 3 (Total for cluster) 
mem_limit
| Ensure it's at least 90% (the default) Note: If you are performing very frequent small inserts, your 2-core nodes may never be able to keep up. Consider grouping your data into larger batches (e.g., once every 30-60 seconds) to reduce the number of files created. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/resource_management/Memory_management.md • administration/management/BE_configuration.md
t
@Rocky all nodes are less than 70% memory usage
r
Even if your memory usage is at 70% (~5.6GB used of 8GB), StarRocks may still be queuing compaction tasks due to per-worker memory safeguards or scheduling thresholds. On an 8GB node, the remaining 2.4GB is very little for a database engine. Here is why your compaction is likely sitting in the queue despite "available" resources: 1. Per-Worker Memory Limit StarRocks has a default limit of 2GB per compaction thread (
compaction_memory_limit_per_worker
). * With 8GB total RAM, if one compaction task estimates it needs 2GB, the system may refuse to start it if the total "committed" memory (used + reserved for tasks) exceeds the safety threshold. * Fix: Force the workers to use less memory so they can fit into your remaining 2.4GB:
Copy code
sql
    -- Reduce per-worker reservation to 1GB or 512MB
    UPDATE information_schema.be_configs SET value = "1073741824" WHERE name = "compaction_memory_limit_per_worker";
2. Check the "Compaction Score" StarRocks won't start a compaction task if the "score" (number of unmerged versions) is too low. * The default 
lake_compaction_score_selector_min_score
is 10. * If your 
MaxCS
(Max Compaction Score) for a partition is 8 or 9, the tasks will stay in the queue or simply not be scheduled, even if the CPU is 100% idle. * Check your scores:
Copy code
sql
    SELECT DATABASE_NAME, TABLE_NAME, PARTITION_NAME, MAX_CS
    FROM information_schema.partitions_meta
    ORDER BY MAX_CS DESC LIMIT 10;
If MaxCS is below 10, the "idleness" is normal behavior. 3. Small Cluster "Concurrency" Math In shared-data mode, the FE calculates the max tasks as 
Nodes * 16
. For your 3-node cluster, that’s 48 tasks. However, with only 2 CPUs per node, having 48 tasks across the cluster is too many. * The FE might be "scheduling" tasks, but the CN nodes are rejecting them because their local queue (
compact_thread_pool_queue_size
) is full or they are waiting for a thread to become free. * Fix: Match the concurrency to your actual hardware.
Copy code
sql
    -- On FE: Reduce global tasks to match your 3 nodes
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "6");

    -- On CN: Match threads to your 2 CPUs
    UPDATE information_schema.be_configs SET value = "1" WHERE name = "compact_threads";
4. Transaction "Publish" Bottleneck If tasks show as "Committed" but not "Finished" in 
SHOW PROC '/compactions'
, the bottleneck isn't CPU or Memory—it's the FE waiting for the metadata version to publish. * If you have a high frequency of small loads (e.g., every 1-2 seconds), the FE's "Publish" queue gets backed up. * Check the Leader FE 
fe.log
for 
publish version timeout
or 
slow publish
. * Solution: Increase your data load interval to at least 15-30 seconds to give the metadata layer room to breathe. Summary Checklist for 8GB Nodes: 1. Lower the worker limit: Set 
compaction_memory_limit_per_worker
to 1GB. 2. Lower the task size: Set 
max_cumulative_compaction_num_singleton_deltas
to 50 (prevents 2GB+ memory spikes). 3. Check Score: Verify if 
MaxCS
is actually $>10$. References • administration/management/BE_configuration.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • faq/shared_data_faq.md References • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md
t
@Rocky the internal 
loads_history
table is heavily compacted
r
The 
_statistics_.loads_history
table (and its corresponding view 
statistics.loads_history
) is an internal system table that tracks every load job in the cluster. In a small cluster with only 2 CPUs, the background compaction of this table can become very noticeable because it competes for the same limited worker threads as your actual data tables. Since every load job creates a new record in this table, frequent small loads (e.g., streaming or high-frequency Flink jobs) cause 
loads_history
to accumulate hundreds of small versions (rowsets) very quickly, triggering constant compaction. Recommendations for a 2-CPU / 8GB Cluster 1. Reduce Sync Frequency (Most Effective) By default, the FE syncs load metadata into this table every 60 seconds. In a resource-constrained cluster, you should increase this interval to batch the writes.
Copy code
sql
-- Increase sync interval from 60s to 5-10 minutes (300-600 seconds)
ADMIN SET FRONTEND CONFIG ("loads_history_sync_interval_second" = "300");
Effect: Reduces the number of small rowsets created in the system table, making compaction tasks occur less frequently. 2. Shorten Retention Period The default retention is 30 days. Keeping 30 days of history on 8GB nodes can lead to large metadata overhead.
Copy code
sql
-- Reduce history retention to 7 days (or even less if not needed for auditing)
ADMIN SET FRONTEND CONFIG ("loads_history_retained_days" = "7");
Effect: Shrinks the physical size of the table, making each compaction task faster and less memory-intensive. 3. Batch Your Data Loads If you are running "Insert Into" or "Stream Load" every few seconds, you are putting immense pressure on the metadata layer of a 2-core node. * Rule of thumb: Aim for load intervals of 30–60 seconds per table. * The compaction you see on 
loads_history
is a symptom of high-frequency "commit" operations. 4. Check Partition/Tablet Count Sometimes the internal 
loads_history
table is created with too many tablets for a small cluster.
Copy code
sql
SHOW TABLET FROM _statistics_.loads_history;
If you see a large number of tablets (e.g., >10), the 2-core BE has to manage compaction for each one. On a 3-node/2-core cluster, you generally want only 1-3 tablets for this internal table. (Note: Modifying system table schemas directly is not recommended; focus on the FE configs above first). Summary The "Heavy Compaction" on this table is normal if you have many loads, but on your hardware, it is likely "starving" your user tables of compaction threads. 1. Increase 
loads_history_sync_interval_second
. 2. Decrease 
loads_history_retained_days
. 3. Decrease the frequency of your actual data ingestions. References • administration/management/FE_configuration.md • loading/loading_introduction/troubleshooting_loading.md • administration/management/compaction.md References • administration/management/FE_configuration.md • loading/loading_introduction/troubleshooting_loading.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md
t
@Rocky compaction scores are improving, some tables are still querying much slower than they used to. Is 6 buckets too many for a data size of about 2MB?
r
Yes, 6 buckets for 2MB of data is too many, especially on a cluster with only 2 CPUs per node. While 6 is the "automatic" default StarRocks calculates for a 3-node cluster (using the formula $2 \times \text{BE\_count}$), it is designed for tables that will eventually hold gigabytes of data. For a 2MB table, this configuration is likely contributing to your performance issues. Why 6 Buckets is Hurting Performance here: 1. Metadata Overhead: Every bucket (tablet) is a physical set of files. With 6 buckets, your 2MB of data is split into chunks of ~333KB. Each of these tiny chunks requires its own metadata tracking in the FE memory and BE thread pools. 2. Thread Contention (The "2 CPU" Problem): When you query a table with 6 buckets, the BE tries to process them in parallel. On a 2-core machine, if you have multiple tables like this, the CPU spends more time "context switching" between dozens of tiny tasks than actually processing data. 3. The 100MB-1GB Rule: StarRocks is optimized for tablets that are 100MB to 1GB (compressed). A 2MB table should ideally be 1 bucket. 4. Compaction Stress: Even with 2MB of data, if you are doing frequent loads, the BE has to run compaction on all 6 tablets independently. This multiplies the background work by 6x for no benefit. Recommendations 1. Set Buckets to 1 for Tiny Tables For any table where you expect the total data size to stay under 100MB-200MB, manually set the bucket count to 1:
Copy code
sql
ALTER TABLE your_table_name DISTRIBUTED BY HASH(join_key) BUCKETS 1;
(Note: In versions before 3.2, you may need to recreate the table to change the bucket count.) 2. Check your "Total Tablet" Count On a node with only 8GB of RAM, having too many total tablets across the entire cluster is a common cause of slowness. * Target: Try to keep the total number of tablets per BE under 1,000–2,000 for a node this small. * Check count:
Copy code
sql
    -- Check how many tablets are on each BE
    SHOW BACKENDS\G
Look at the 
TabletNum
column. If it's in the many thousands, you need to consolidate your small tables into fewer buckets or use dynamic partitioning to drop old, tiny partitions. 3. Use "Single Tablet Parallelism" If you reduce to 1 bucket and fear the query will be slow because it only uses one thread, ensure this session variable is on:
Copy code
sql
SET GLOBAL enable_tablet_internal_parallel = true;
This allows StarRocks to use multiple threads to scan a single tablet, giving you the best of both worlds: low metadata overhead and parallel execution. Summary For a 2MB table on 2-core nodes, 1 bucket is almost always the correct choice. The "6 buckets" you are seeing is the cluster trying to be "ready for scale," but on your hardware, that extra management work is slowing you down. References • knowledge/usage/best-practice.md • table_design/data_distribution/Data_distribution.md • best_practices/query_tuning/schema_tuning.md • faq/operation_maintenance_faq.md References • knowledge/usage/best-practice.md • table_design/data_distribution/Data_distribution.md • faq/operation_maintenance_faq.md • best_practices/query_tuning/schema_tuning.md • best_practices/bucketing.md
t
@Rocky 2MB per partition not the entire table
r
Even with 2MB per partition, 6 buckets is still significantly over-provisioned. At that size, each bucket (tablet) holds only about 333KB of data. On a cluster with only 2 CPUs per node, this creates a specific performance bottleneck known as "metadata amplification." Here is why this is slowing down your queries and how to fix it: 1. The Tablet Explosion Problem If you have a table with 6 buckets per partition and you have, for example, 100 partitions (e.g., 100 days of data), that single table has 600 tablets. * The Overhead: Every time you run a query, the FE must plan the execution for 600 separate objects, and the BE must open and manage files for all of them. * On 2-core nodes: Your CPUs are likely spending more time "scheduling" the work across these hundreds of tablets than actually reading the 2MB of data. 2. Compaction Efficiency Compaction runs at the tablet level. * With 6 buckets, the system has to run 6 separate compaction tasks just to clean up one partition. * Because each task handles only 333KB, the "fixed cost" of starting a compaction task (CPU cycles, disk I/O for metadata) outweighs the actual work being done. This is likely why you see your system heavily occupied with internal tasks. 3. Recommendations for 2MB Partitions A. Reduce to 1 Bucket For partitions that are only 2MB, you should use 1 bucket. Even 1 bucket is "small" by StarRocks standards (which prefers 100MB+), but it reduces your management overhead by 6x.
Copy code
sql
-- Change future partitions to use 1 bucket
ALTER TABLE your_table_name SET ("default.replication_num" = "3"); -- (Ensure replication is correct)
ALTER TABLE your_table_name DISTRIBUTED BY HASH(your_column) BUCKETS 1;
B. Re-evaluate Partitioning Strategy If your partitions are only 2MB, you might be over-partitioning. _ If you are partitioning by hour, consider switching to _day*. _ If you are partitioning by _day*, and the total table size is only a few hundred MBs, consider removing partitioning entirely and just using a single bucketed table. _ _Rule of thumb:* A partition should ideally hold at least 100MB-1GB of data to justify the overhead of the partition metadata. C. Check the "Total Tablet" Limit On 8GB/2-core nodes, the "hidden killer" is the total number of tablets across the whole BE. * Run 
SHOW PROC '/statistic';
to see the total number of tablets in your cluster. * If that number is over 2,000–3,000, your 2-core CPUs will struggle with background "heartbeat" and metadata tasks, causing the slowness you are experiencing. Summary For 2MB partitions, 6 buckets is 6x too many. By reducing to 1 bucket, you reduce the number of files the BE has to open and the number of tasks the FE has to schedule, which will directly improve your query latency on small-core hardware. References • table_design/data_distribution/Data_distribution.md • knowledge/usage/best-practice.md • best_practices/bucketing.md • administration/management/compaction.md
t
@Rocky these are shared data tables, does replication factor still apply?
r
In a shared-data (Cloud Native) architecture, the 
replication_num
property does not apply to data files in the same way it does for shared-nothing tables. Here is how data durability and availability work in shared-data mode: 1. Remote Storage is the "Replica" In shared-data mode, StarRocks stores only one copy of the data files on your remote storage (S3, GCS, HDFS, etc.). You do not need to set a replication factor because the underlying storage system (e.g., AWS S3) already provides extremely high durability (99.999999999%) and availability by replicating the data across multiple physical disks and availability zones internally. 2. Tablets in Shared-Data While the concept of "tablets" and "buckets" still exists for shared-data tables, they work differently: * Shared-Nothing: A tablet is a physical directory on a specific BE's local disk. 
replication_num = 3
means 3 different BEs each have a physical copy. * Shared-Data: A tablet is a logical metadata object. The actual data files reside in object storage. Any Compute Node (CN) or BE can technically process any tablet because they all pull from the same central bucket. 3. What happens if you specify 
replication_num
? In most shared-data implementations: * The property is ignored or defaults to 1. * If you try to set 
replication_num = 3
on a cloud-native table, the system usually defaults it back to 1 because there is no mechanism (or need) to store three identical copies of the same file in your S3 bucket. 4. Local Cache (Data Cache) Instead of replicas, shared-data uses Local Disk Cache. * When a query runs, the BE/CN downloads data from S3 to its local disk. * This is not a "replica" for durability; it is a "cache" for speed. * If a BE node dies, no data is lost because the master copy is in S3. Another BE simply takes over the tablet and fetches the data from S3. Summary for your 2-CPU Cluster Since you are using shared-data, your focus should remain on reducing the number of buckets (as discussed before) and managing your Data Cache (
datacache.enable
and 
datacache.partition_duration
). You don't need to worry about `replication_num`; your hardware is already "off the hook" for the CPU and I/O overhead of maintaining multiple synchronous data copies. References • administration/management/resource_management/Replica.md • deployment/shared_data/s3.md • faq/shared_data_faq.md References • administration/management/resource_management/Replica.md • administration/management/FE_configuration.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • faq/shared_data_faq.md • deployment/shared_data/s3.md
t
@Rocky how to set data cache configs
r
In a shared-data (Cloud Native) cluster, Data Cache settings are primarily managed through the Compute Node (CN) or Backend (BE) configuration files (
cn.conf
or 
be.conf
) and occasionally via table properties. 1. Enable and Locate the Cache Data Cache is enabled by default in StarRocks v3.2.3 and later. * 
starlet_use_star_cache
: Set to 
true
to enable the cache. * 
storage_root_path
: Defines where the cache is stored on your local disk. * Example: 
storage_root_path = /data1/storage;/data2/storage
* The cache files will specifically be stored in 
<storage_root_path>/starlet_cache
. 2. Control Cache Size You can limit how much disk and memory the cache consumes to prevent it from starving other processes. * 
starlet_star_cache_disk_size_percent
: The maximum percentage of the disk (specified in 
storage_root_path
) that the cache can use. Default is 80. * 
datacache_disk_size
: You can also set a hard limit in bytes (e.g., 
2147483648
for 2GB). The system will take the greater value between this and the percentage setting. * 
datacache_mem_size
: Sets the memory limit for the "Metadata" and "Page Cache" portions of the data cache. 3. Table-Level Overrides You can control caching behavior for specific tables or partitions when creating or altering them: * 
datacache.enable
: Set to 
true
(default) or 
false
. Useful if you have a massive "cold" table you don't want to wipe out the cache for more important data. * 
datacache.partition_duration
: Defines how long data from a partition remains in the cache. * Example: 
("datacache.partition_duration" = "7 days")
means only the last 7 days of data will be prioritized for caching. 4. How to Verify After changing these in 
cn.conf
and restarting the node, you can verify the status:
Copy code
sql
-- Check DiskUsage and MemUsage columns
SHOW BACKENDS\G

-- Or query the system metrics directly
SELECT * FROM information_schema.be_datacache_metrics;
Important for your 2-CPU/8GB Setup: On small nodes, be careful with 
datacache_mem_size
. If you set it too high, the BE process might be killed by the OS Out-of-Memory (OOM) killer. Since you only have 8GB of RAM, I recommend leaving the memory settings at their defaults and focusing on ensuring you have enough fast local SSD space for the 
storage_root_path
. References • using_starrocks/caching/block_cache.md • deployment/shared_data/s3.md • data_source/data_cache_observe.md • administration/management/BE_configuration.md • deployment/shared_data/feature-support-shared-data.md References • data_source/data_cache_observe.md • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_compaction.md
t
@Rocky when I look at the compute nodes I see their disk cache metrics seem fairly low, is this normal? DataCacheMetrics: Status: Normal, DiskUsage: 0B/0B, MemUsage: 73.5MB/1.6GB
r
No, 
DiskUsage: 0B/0B
is not normal for a shared-data cluster that you want to be performant. It means your disk-based Data Cache (StarCache) is unconfigured or disabled. In a healthy setup, the denominator should show your total allocated cache space (e.g., 
50GB
), and the numerator will grow as you query data. Why you are seeing 0B/0B When the denominator is 
0B
, it means the Compute Node (CN) hasn't identified any local storage to use for caching. This usually happens for one of three reasons: 1. 
storage_root_path
is not set If the 
storage_root_path
in your 
cn.conf
is empty or pointing to a non-existent directory, the cache cannot initialize. * Fix: Ensure 
storage_root_path
points to a valid local directory (ideally an SSD).
Copy code
properties
    # cn.conf
    storage_root_path = /home/starrocks/cn/storage
2. 
starlet_use_star_cache
is set to 
false
 In versions older than v3.2.3, this was sometimes 
false
by default. If this is 
false
, StarRocks will not initialize the disk cache system at all. * Fix: Explicitly enable it in 
cn.conf
.
Copy code
properties
    starlet_use_star_cache = true
3. Quota is set to 0 If you have set 
datacache_disk_size = 0
or if 
starlet_star_cache_disk_size_percent
is somehow 0, the system will allocate zero bytes. * Fix: Check these parameters in 
cn.conf
. By default, 
starlet_star_cache_disk_size_percent
is 80, meaning it uses 80% of the disk where 
storage_root_path
resides. Why 
MemUsage
is 73.5MB The 
MemUsage
you see is likely just the metadata cache (the index that keeps track of what should be in the cache) or a very small "Page Cache." Since the disk cache isn't working, this memory isn't doing much to speed up your actual data scans. How to Fix and Verify 1. Update 
cn.conf
:
Copy code
properties
    storage_root_path = /your/fast/ssd/path
    starlet_use_star_cache = true
    # Optional: Hard limit of 20GB if you don't want to use the 80% default
    # datacache_disk_size = 21474836480
2. Restart the CN node. 3. Run a query: Query a few tables. The cache is "lazy," meaning it only fills up when you actually scan data. 4. Check again: Run 
SHOW BACKENDS\G
. You should now see a denominator larger than 
0B
. Note: If you are running on Kubernetes, ensure your CN pods have a 
PersistentVolume
(PV) or 
hostPath
mounted at the 
storage_root_path
. If the path is just a temporary container layer, it may show 0 or cause high I/O wait. References • using_starrocks/caching/block_cache.md • deployment/shared_data/s3.md • data_source/data_cache_observe.md • administration/management/BE_configuration.md • deployment/shared_data/feature-support-shared-data.md References • using_starrocks/caching/block_cache.md • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/minio.md
t
@Rocky is it possible to change during runtime, otherwise how to make this change using the k8s operator
r
No, it is not possible to change the 
storage_root_path
during runtime. Since it defines where the physical cache files are stored, it is a static configuration that requires a process restart to take effect. In a Kubernetes environment managed by the StarRocks Operator, you should make this change in your 
StarRocksCluster
Custom Resource (CRD). When you update the CRD, the Operator will automatically perform a rolling restart of your Compute Nodes (CN) to apply the settings. How to apply via K8s Operator To fix the 
0B/0B
issue, you must define a storage volume and then point the StarRocks config to it. 1. Configure the YAML Add the 
storageSpec
to create a persistent volume and the 
config
section to enable the cache in your 
StarRocksCluster
YAML:
Copy code
yaml
apiVersion: <http://starrocks.com/v1alpha1|starrocks.com/v1alpha1>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  starrocksComputeNodesSpec:
    replicas: 2
    # 1. Define the physical storage for the cache
    storageSpec:
      name: cn-cache
      storageClassName: standard  # Or your specific SSD storage class
      storageSize: 50Gi
      mountPath: /opt/starrocks/cn/storage
    # 2. Tell StarRocks to use that path and enable the cache
    config: |
      storage_root_path = /opt/starrocks/cn/storage
      starlet_use_star_cache = true
2. Apply the change
Copy code
bash
kubectl apply -f starrocks-cluster.yaml
Why this is necessary on K8s * Automatic Restart: The Operator detects the change in the 
config
or 
storageSpec
and restarts the pods one by one. * Mounting: Without the 
storageSpec
(specifically the 
mountPath
), the CN process might look for 
/opt/starrocks/cn/storage
inside the container's ephemeral layer. If that layer has no size limit defined or is restricted, the cache may fail to initialize or show as 
0B
. * Persistence: Using a 
PersistentVolume
(via 
storageSpec
) ensures that if a pod restarts, it doesn't have to re-download the entire cache from S3, which saves you significant egress costs and time. Verification Once the rolling restart is complete, check the metrics again:
Copy code
sql
SHOW BACKENDS\G
-- DiskUsage should now show something like: 0B / 40GB (assuming 80% of 50GB)
The numerator (
0B
) will stay at zero until you run a query that fetches data from your remote storage. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/shared_data/minio.md • quick_start/helm.md References • administration/management/BE_configuration.md • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md • quick_start/helm.md
t
@Rocky anyway to check this config at runtime?
r
Yes, you can check the active configuration of your Compute Nodes (CN) or Backends (BE) at runtime using either SQL or an HTTP endpoint. This is the best way to verify if your 
cn.conf
changes were actually picked up. 1. Using SQL (Recommended) You can query the 
information_schema.be_configs
table. This table aggregates configurations from all nodes in the cluster.
Copy code
sql
SELECT * 
FROM information_schema.be_configs 
WHERE NAME IN ('storage_root_path', 'starlet_use_star_cache', 'datacache_disk_size');
* 
VALUE
: Shows the current active value. * 
MUTABLE
: If 
false
, the parameter requires a restart to change. Both 
storage_root_path
and 
starlet_use_star_cache
are typically not mutable at runtime. 2. Using HTTP API If you cannot access the SQL interface or want to check a specific node directly, you can use the 
/varz
endpoint on the CN's web port (default 
8040
):
Copy code
bash
# Replace <CN_IP> with your pod IP or service address
curl http://<CN_IP>:8040/varz | grep -E 'storage_root_path|starlet_use_star_cache'
3. Check effective Disk Capacity Since your metrics showed 
0B/0B
, you can also check what the system thinks its total capacity is by running:
Copy code
sql
SHOW BACKENDS\G
Look for the 
DataCacheMetrics
column. If it still says 
0B/0B
after you've applied the config and restarted, it means the CN process failed to initialize the directory at 
storage_root_path
(usually due to permissions or the path not existing inside the container). Summary of common states: * 
starlet_use_star_cache = false
: Data cache is completely disabled; no metrics will show. * 
storage_root_path
is empty: Disk cache cannot initialize; will show 
0B/0B
. * 
storage_root_path
is valid: You will see the total size (e.g., 
0B/40GB
). References • sql-reference/information_schema/be_configs.md • administration/management/BE_configuration.md • data_source/data_cache_observe.md References • sql-reference/information_schema/be_configs.md • administration/management/BE_configuration.md • data_source/data_cache_observe.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md • deployment/deploy_manually.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md
t
@Rocky provided yaml is incorrect
r
The error in the previous YAML was likely the field name. In the 
StarRocksCluster
Custom Resource (CRD) used by the operator, the correct key is 
starRocksCnSpec
 (not 
starrocksComputeNodesSpec
), and persistence is usually configured via the 
storageVolumes
 array. Here is the corrected YAML for the StarRocks Operator:
Copy code
yaml
apiVersion: <http://starrocks.com/v1alpha1|starrocks.com/v1alpha1>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  starRocksCnSpec:
    replicas: 3
    # 1. Provide the configuration for the cache
    config: |
      storage_root_path = /opt/starrocks/cn/storage
      starlet_use_star_cache = true
    # 2. Define the Persistent Volume Claim (PVC) to be mounted
    storageVolumes:
      - name: cn-cache-storage
        storageClassName: "standard" # Replace with your SSD storage class
        storageSize: 50Gi
        mountPath: /opt/starrocks/cn/storage
Key Corrections: * Field Name: Changed 
starrocksComputeNodesSpec
to 
starRocksCnSpec
. * Storage Structure: Changed 
storageSpec
to 
storageVolumes
, which is an array. This is the standard way to map a PVC to a specific 
mountPath
in the StarRocks Pods. * Path Alignment: The 
mountPath
in the volume definition must exactly match the 
storage_root_path
in the 
config
block. If you are using the Helm Chart If you are managing your cluster via Helm (editing 
values.yaml
), the structure is slightly different because the chart abstracts the CRD. In 
values.yaml
, it usually looks like this:
Copy code
yaml
starrocksCnSpec:
  config: |
    storage_root_path = /opt/starrocks/cn/storage
    starlet_use_star_cache = true
  storageSpec: # Helm chart often uses 'storageSpec' to simplify the CRD's 'storageVolumes'
    name: "cn-storage"
    storageSize: 50Gi
    storageClassName: "standard"
    mountPath: /opt/starrocks/cn/storage
Verification After applying the change (
kubectl apply -f ...
), the Operator will perform a rolling restart. You can verify the volume is mounted by checking the pod description:
Copy code
bash
kubectl describe pod <cn-pod-name>
Look for the Mounts section to ensure 
/opt/starrocks/cn/storage
is attached to a Persistent Volume and not an 
emptyDir
. References • administration/management/BE_configuration.md • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md • quick_start/helm.md • deployment/shared_data/azure.md • deployment/shared_data/gcs.md • deployment/shared_data/feature-support-shared-data.md • sql-reference/information_schema/be_configs.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/FE_configuration.md • deployment/deploy_manually.md
t
@Rocky looking at the configs, cache is enabled
r
If you have verified that 
starlet_use_star_cache = true
is in your configuration but still see 
0B / 0B
in 
SHOW BACKENDS
, the most common reason is a mismatch between the container's physical storage and the configuration path. In Kubernetes, the StarRocks process cannot use the cache if the directory defined in 
storage_root_path
does not exist or has no disk space allocated via a Persistent Volume. Correct StarRocksCluster YAML Use this structure for the StarRocks Operator. Ensure the 
mountPath
in 
storageVolumes
exactly matches the 
storage_root_path
in the 
config
.
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  starRocksCnSpec:  # Note the camelCase
    replicas: 3
    image: starrocks/cn-ubuntu:3.3-latest
    # 1. Define the parameters
    config: |
      storage_root_path = /opt/starrocks/cn/storage
      starlet_use_star_cache = true
      starlet_star_cache_disk_size_percent = 80
    # 2. Mount physical storage to that path
    storageVolumes:
      - name: cn-cache-volume
        storageClassName: "your-ssd-class" # Replace with your storage class
        storageSize: 100Gi
        mountPath: /opt/starrocks/cn/storage
Why it might still show 0B / 0B If the YAML is applied and pods have restarted, but you still see zero capacity: 1. Path Mismatch: Ensure 
storage_root_path
in the 
config
block is exactly the same as the 
mountPath
in 
storageVolumes
. If they differ, StarRocks looks for the cache on the container's ephemeral root partition, which often reports 0 available space for the cache engine. 2. Mount Verification: Run 
kubectl describe pod <cn-pod-name>
. Check the Mounts section. You should see: * 
/opt/starrocks/cn/storage
mounted from 
cn-cache-volume (rw)
. * If you don't see this mount, the Operator failed to update the Pod spec. 3. Check Effective Config: Run this SQL to see what the node actually thinks its config is:
Copy code
sql
    SELECT * FROM information_schema.be_configs
    WHERE NAME IN ('storage_root_path', 'starlet_use_star_cache');
4. Subdirectory Creation: StarRocks automatically creates a 
/starlet_cache
folder inside your 
storage_root_path
. If the process doesn't have permission to write to the mounted volume, it will fail to initialize the cache and report 0B. Note on Helm If you are using the Helm Chart instead of raw CRD YAMLs, the configuration is often nested differently in `values.yaml`:
Copy code
yaml
starrocksCnSpec:
  storageSpec: # Helm often simplifies 'storageVolumes' to 'storageSpec'
    enabled: true
    storageSize: 100Gi
    mountPath: /opt/starrocks/cn/storage
Next Step: Verify the mount inside the pod using 
kubectl exec <pod> -- df -h
to see if 
/opt/starrocks/cn/storage
is indeed a separate disk with the expected size. References • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md
t
@Rocky 
storage_root_path
and volume is mounted correctly with read-write permissions
r
If you have verified that 
storage_root_path
is correctly mounted with RW permissions and the toggle is enabled, but you still see 
0B / 0B
capacity, the issue is likely how StarRocks is calculating the available space or a version-specific configuration requirement. 1. Fix the Capacity Calculation By default, StarRocks uses a percentage of the disk (
starlet_star_cache_disk_size_percent
defaults to 80%). In some Kubernetes environments, the system may fail to detect the total disk capacity of the volume, resulting in 
80% of 0 = 0B
. The Fix: Manually set an absolute size instead of a percentage. Add this to your `config`:
Copy code
properties
# Set this to slightly less than your PVC size (e.g., 45G for a 50G PVC)
datacache_disk_size = 45G
The system takes the greater of 
datacache_disk_size
and the percentage-based calculation. 2. Verify both Enable Toggles Depending on your StarRocks version, the cache requires specific flags. Ensure both of these are checked in `information_schema.be_configs`: * `starlet_use_star_cache`: Must be 
true
. * `datacache_enable`: Must be 
true
(In versions 3.3+, this defaults to true, but in older versions, it may need to be explicit). 3. Check for the 
starlet_cache
Subdirectory StarRocks does not store cache files directly in the root of 
storage_root_path
. It creates a subdirectory: 
<storage_root_path>/starlet_cache/
If the StarRocks process (user 
starrocks
or 
root
inside the container) doesn't have permission to create a new directory inside your mount point, the cache will fail to initialize. * Test: 
kubectl exec -it <pod> -- mkdir -p /opt/starrocks/cn/storage/starlet_cache
* If this fails, your mount has permission issues despite appearing RW. 4. Inspect the CN Logs (The most definitive step) If the cache fails to initialize, the reason will be explicitly logged during the CN startup. Search the logs for "cache" or "starlet":
Copy code
bash
kubectl logs <cn-pod-name> | grep -iE "cache|starlet"
Look for errors like: * 
fail to init star cache
* 
starlet_cache_path not exist
* 
capacity is 0, disable star cache
Summary Checklist for your YAML:
Copy code
yaml
config: |
  storage_root_path = /opt/starrocks/cn/storage
  starlet_use_star_cache = true
  datacache_enable = true
  datacache_disk_size = 45G
References • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • deployment/shared_data/s3.md • data_source/data_cache_observe.md References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md
t
@Rocky I ran 
UPDATE <http://information_schema.be|information_schema.be>_configs SET value = "45G" WHERE name = "datacache_disk_size";
and still don't see a difference
r
Updating 
datacache_disk_size
via SQL is the correct method for dynamic scaling, but if it still shows 
0B / 0B
in 
SHOW BACKENDS
, it usually means the Data Cache engine failed to initialize at startup or is currently disabled. Here is how to troubleshoot and fix this: 1. Verify if the config actually updated The 
UPDATE be_configs
statement only affects the running process in memory. First, confirm that the node acknowledges the new value:
Copy code
sql
ADMIN SHOW CONFIG LIKE '%datacache_disk_size%';
If the value is still 
0
or different from what you set, your 
UPDATE
statement might have missed a specific node. Try applying it to all:
Copy code
sql
UPDATE information_schema.be_configs SET value = "45G" WHERE name = "datacache_disk_size";
2. Check the "Greater Of" Logic In shared-data (CN) clusters, StarRocks determines cache size by taking the greater of two parameters: * 
datacache_disk_size
(Absolute value) * 
starlet_star_cache_disk_size_percent
(Percentage of disk, default 80%) If the system cannot detect the disk capacity of your mount, the percentage calculation returns 
0
. Even if you set 
datacache_disk_size
, the cache engine may remain "Disabled" if it failed to start initially. 3. Check for Engine Initialization Errors (Most likely) If the engine is stuck at 
0B
, it usually means it encountered an error when trying to create the cache files in your 
storage_root_path
. Check the CN logs immediately:
Copy code
bash
kubectl logs <cn-pod-name> | grep -iE "cache|starlet"
Look for these specific errors: * "no writable storage path": The path exists, but the 
starrocks
user inside the container doesn't have permissions. * "starlet_cache_path not exist": There is a mismatch between your 
storage_root_path
and the actual mount. * "capacity is 0, disable star cache": The engine gave up because it couldn't find usable space. 4. Persistence and YAML Since you are using Kubernetes, any 
UPDATE be_configs
will be wiped out if the pod restarts. You must ensure your 
StarRocksCluster
YAML or Helm 
values.yaml
is updated: Correct YAML Structure:
Copy code
yaml
spec:
  starRocksCnSpec:
    config: |
      storage_root_path = /opt/starrocks/cn/storage
      starlet_use_star_cache = true
      datacache_disk_size = 45G
    storageVolumes:
      - name: cn-storage
        mountPath: /opt/starrocks/cn/storage
        storageSize: 50Gi
5. Check the version * v3.3 and below: You may need to set 
starlet_star_cache_disk_size_bytes
instead of 
datacache_disk_size
. * v3.4 and above: The unified 
datacache_disk_size
is the standard. Next Step: If logs show no errors but it’s still 
0B
, try a manual restart of the CN pods after ensuring the YAML contains 
datacache_disk_size = 45G
. This forces the cache engine to re-initialize with the new quota. References • data_source/data_cache.md • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • deployment/deploy_manually.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md
• sql-reference/sql-statements/data-manipulation/UPDATE.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md • sql-reference/information_schema/be_configs.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md • data_source/data_cache_observe.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md • deployment/shared_data/hdfs.md • deployment/shared_data/azure.md • deployment/shared_data/gcs.md • deployment/shared_data/feature-support-shared-data.md • quick_start/helm.md • administration/management/FE_configuration.md References • data_source/data_cache.md • administration/management/BE_configuration.md
Open in Slack
PreviousNext
Powered by