http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> What are some configuration options...
# questions-and-troubleshooting
b
@Rocky What are some configuration options in a fresh deployed StarRocks shared data cluster I should consider setting/changing? I am specifically interested in caching, compaction, optimizations, etc
r
Hey there! I'm Rocky. Setting up a fresh StarRocks shared-data cluster (compute-storage separation) is an exciting step. Because the data lives on remote object storage (S3, GCS, Azure Blob, etc.), performance tuning shifts heavily toward caching, compaction, and garbage collection. Here are the key configuration options you should consider: 1. Data Cache (Crucial for Performance) In shared-data mode, the Data Cache on your BE/CN nodes is what prevents every query from having to fetch data from remote storage. * 
starlet_use_star_cache
(BE Config): In versions 3.2.3+, this is enabled by default. If you are on an older version, ensure this is set to 
true
. * 
starlet_star_cache_disk_size_percent
(BE Config): This defines how much of your BE's local disk is used for caching. The default is 90%. If your disks are shared with other processes, you might want to tune this. * 
datacache.partition_duration
(Table Property): This is a powerful setting. It defines the time range of "hot" data to keep in the cache. If your queries target the last 7 days of data, set this accordingly so older data doesn't evict the hot data. * Data Cache Warmup: Use the 
CACHE SELECT
command (introduced in v3.3.0) to proactively pull data into the cache before users start querying it. 2. Compaction (Optimizing File Layout) Compaction merges small files into larger ones to reduce I/O overhead. In shared-data mode, the FE schedules compaction, and the BE executes it. * 
compact_threads
(BE Config): Controls how many compaction tasks a BE can run. For production, it is recommended to set this to roughly 25% of your CPU cores. * 
lake_compaction_max_tasks
(FE Config): Controls the total number of concurrent compaction tasks across the cluster. The default 
-1
lets StarRocks calculate it based on BE count, but you can manually cap it if compaction is consuming too many resources. * 
max_cumulative_compaction_num_singleton_deltas
(BE Config): It is often recommended to reduce this to 
100
(from the default 1000) to ensure compaction tasks finish faster and release memory more quickly. * 
lake_enable_vertical_compaction_fill_data_cache
(BE Config): Set this to 
true
if you want compaction results to be immediately cached. This is very helpful if your ingestion and queries happen in different warehouses. 3. Vacuum and Garbage Collection Since every update creates a new version in object storage, you need to clean up old versions to save costs. * 
lake_autovacuum_grace_period_minutes
(FE Config): This defines how long to keep historical versions. The default is 
30
. Important: Ensure this value is longer than your longest-running query to prevent data from being deleted while a query is still reading it. * 
lake_autovacuum_parallel_partitions
(FE Config): If you have a massive number of partitions and storage costs are creeping up, increase this (default is 
8
) to speed up the cleanup process. 4. Query & Storage Optimizations * Tablet Sizing: Aim for 1GB to 5GB of data per tablet. Too many small tablets lead to metadata overhead; too few tablets limit parallelism. * Spill to Disk: If you run complex queries that exceed memory, consider setting the session variable 
enable_spill = true
to allow queries to use local disk space instead of failing with an "Out of Memory" error. * Pipeline DOP: Ensure 
pipeline_dop
is set to 
0
(default), which allows StarRocks to automatically scale parallelism based on your BE core count. How to monitor: You can check the health of your compaction and caching with these commands: * 
SHOW PROC '/compactions';
_ 
SELECT _ FROM information_schema.be_cloud_native_compactions;
* Check the Query Profile for metrics like 
CompressedBytesReadRemote
vs. 
CompressedBytesReadLocal
to see your cache hit rate.
Do you have a specific workload (like real-time ingestion or heavy analytics) you're optimizing for? I can give more tailored advice! References • administration/management/FE_configuration.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • knowledge/admin_and_operation/shared_data_compaction.md
b
@Rocky I have these options set on my compute nodes
Copy code
enable_spill = true
    datacache_enable = true
    datacache_mem_size = 40%
    datacache_disk_size = 75%
    query_pool_spill_mem_limit_threshold = 0.8
    query_cache_capacity = 4294967296
    primary_key_limit_size = 512
    tablet_writer_rpc_timeout_sec = 600
    write_buffer_size = 209715200
    brpc_max_body_size = 4294967296
    max_cumulative_compaction_num_singleton_deltas = 100
And these options set on my frontend nodes
Copy code
catalog_trash_expire_second = 3600
    enable_query_cache = true
    run_mode = shared_data
    enable_load_volume_from_conf = false
    brpc_connection_pool_size = 16
Can you suggest anything else? I have very large complex joins and occasionally do large table backfills. Is there a way to run compaction out of standard usage hours too?
r
That’s a solid baseline configuration for a shared-data (compute-storage separation) cluster. Since you are dealing with very large complex joins and backfills, here are some additional suggestions to improve stability and performance, along with the answer to your compaction question. 1. Recommendations for Large Joins & Backfills For large joins, the priority is managing memory and ensuring the "spill to disk" mechanism is efficient. For backfills, the priority is managing the write pipeline and memory limits. Compute Node (CN/BE) Suggestions: * 
spill_local_storage_dir
: Ensure this is explicitly set to a high-performance disk (SSD/NVMe). If possible, use a different disk from your data cache to avoid I/O contention during heavy joins. * 
load_process_max_memory_limit_percent = 50
: (Default is 30%). For large backfills, increasing the percentage of memory allocated to the loading process can prevent 
Memory Limit Exceeded
errors during the write/merge phase. * 
lake_enable_vertical_compaction_fill_data_cache = true
: In shared-data mode, this ensures that data merged during compaction is filled into the local cache. This is critical if your backfills are immediately followed by queries on that same data. * 
thrift_server_max_worker_threads = 1000
: If you are running many parallel backfill jobs (e.g., many Stream Loads), increasing this helps handle the connection overhead. Frontend (FE) Suggestions: * 
max_query_timeout = 3600
: (Default 300s). Complex joins on massive datasets often need more than 5 minutes. You can set this globally or per session. * 
lake_compaction_max_tasks = -1
: (You likely have this at default, but check). This allows the FE to automatically scale compaction tasks based on the number of CNs. ────────── 2. Running Compaction Out of Standard Hours In StarRocks shared-data mode, compaction is scheduled by the FE and executed by the CNs. There isn't a native "time-of-day" scheduler inside the FE configuration, but you can achieve this in two ways: Option A: External Scheduling (Recommended) You can use a simple cron job on a management node that executes SQL commands to enable/disable the FE's compaction scheduler. * To disable compaction (at the start of peak hours):
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
* To enable compaction (at the start of off-peak hours):
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");
Option B: Manual Compaction (v3.1+) If you want to target specific large tables after a backfill, you can trigger compaction manually during your maintenance window:
Copy code
sql
ALTER TABLE <table_name> COMPACT;
Tuning Compaction Intensity If you choose to run it during off-peak hours, you can make it more aggressive by increasing the threads on your CNs: * 
compact_threads
: Default is 4. For off-peak maintenance, you can increase this to 25%–50% of your CPU cores to finish the backlog faster.
Copy code
sql
    -- Example for CN nodes
    UPDATE information_schema.be_configs SET value = "16" WHERE name = "compact_threads";
Summary of suggested additions:
Copy code
properties
# CN/BE Configs
load_process_max_memory_limit_percent = 50
lake_enable_vertical_compaction_fill_data_cache = true
compact_threads = 8  # Increase if backfills cause version bloat

# FE Configs
max_query_timeout = 3600
If your backfills are using 
INSERT INTO SELECT
, ensure you also set 
SET enable_spill = true;
for those specific sessions to handle the massive join/shuffle memory requirements. References • knowledge/usage/best-practice.md • loading/loading_introduction/loading_considerations.md • loading/loading_introduction/troubleshooting_loading.md
b
@Rocky I can configure a separate disk for my compute nodes spill as well? Does enabling spill have any downsides other than stopping OOMs and making queries that use it slower?
r
Yes, you can and absolutely should configure a separate disk for spilling on your Compute Nodes (CN) or Backend nodes (BE). Configuring Separate Disks You can specify one or more local directories for spilling by modifying the 
cn.conf
(for CN) or 
be.conf
(for BE) file using the 
spill_local_storage_dir
parameter:
Copy code
properties
# Example: Using two separate dedicated SSDs for spilling
spill_local_storage_dir=/mnt/ssd1/spill;/mnt/ssd2/spill
Note: After changing this configuration, you must restart the node for it to take effect. Downsides of Enabling Spill While spilling is a lifesaver for preventing OOMs in memory-intensive tasks (like large joins, sorts, or building Materialized Views), there are several trade-offs: 1. Significant Latency Increase: Queries that trigger spilling typically see a tenfold (10x) or greater increase in latency compared to running purely in-memory. You will likely need to increase your 
query_timeout
session variable for these queries. 2. High Disk I/O & Load: Spilling generates a massive amount of write and read operations. If you don't use a separate disk, this I/O contention can slow down all other operations on that node, including standard data scans. 3. No Per-Query Resource Isolation: The 
spill_local_storage_dir
is shared by all concurrent queries on a node. Currently, StarRocks does not support a "per-query" spill disk quota, meaning one massive query could potentially exhaust the spill disk space and cause other concurrent spilling queries to fail. 4. Limited Scope: Spilling doesn't solve every OOM. It currently supports Aggregate, Sort, Join, and CTE operators. It cannot release memory used for expression evaluation or other internal metadata structures. 5. Disk Space Management: While StarRocks automatically clears spill data after a query finishes (or upon restart if it crashed), a very large query can quickly fill up a disk if the disk is not sized correctly for your ETL workloads. Rocky's Tip: In production, always use high-performance disks (like NVMe SSDs) for your spill directory to minimize the performance penalty. If you are in a cloud environment, you can also enable spilling to object storage (S3/GCS/OSS) after the local disk reaches capacity to provide an extra safety net. References • administration/management/resource_management/spill_to_disk.md • knowledge/usage/best-practice.md • sql-reference/System_variable.md • administration/management/BE_configuration.md
b
@Rocky Which of these are system vars and which go in configurations?
r
In StarRocks, the distinction between System Variables and Configuration Parameters depends on how they are managed and their scope of influence. Since you didn't provide a specific list, here is the general rule to help you categorize them, along with common examples for each: 1. System Variables These control the behavior of the SQL engine, query execution, and session-level settings. * How to set them: Use the 
SET
command (e.g., 
SET query_mem_limit = 2147483648;
). * How to view them: Use 
SHOW VARIABLES;
. * Scope: Can be Global (affects new sessions), Session (current connection only), or used as a Hint (
/_+ SET_VAR(pipeline_dop=8) _/
) for a single query. * Common Examples: * 
query_mem_limit
* 
pipeline_dop
* 
wait_timeout
* 
storage_engine
* 
character_set_server
2. Configuration Parameters (FE/BE) These control the underlying infrastructure, server ports, logging, and background maintenance tasks. * How to set them: Modified in the 
fe.conf
or 
be.conf
files. Some can be changed dynamically using 
ADMIN SET FRONTEND CONFIG
or 
ADMIN SET BACKEND CONFIG
. * How to view them: Use 
ADMIN SHOW FRONTEND CONFIG;
or 
ADMIN SHOW BACKEND CONFIG;
. * Scope: System-wide or Node-wide. These usually require a cluster admin to change and often affect how the process (FE or BE) starts and communicates. * Common Examples: * 
be_port
/ 
http_port
* 
priority_networks
* 
sys_log_level
* 
tablet_writer_open_timeout
* 
storage_root_path
────────── Quick Rule of Thumb: * If you can change it with 
SET
, it is a System Variable. * If you have to edit a 
.conf
file or use 
ADMIN SET CONFIG
, it is a Configuration Parameter. If you have a specific list you're looking at, feel free to paste it here and I can categorize them for you! References • administration/management/FE_configuration.md • administration/management/BE_configuration.md
b
I have these ones @Rocky
Copy code
set global enable_query_cache = true;
set global enable_runtime_adaptive_dop = true;
set global pipeline_dop = 0;
set global enable_profile = false;
set global enable_pipeline_engine = true;
set global parallel_fragment_exec_instance_num = 1;
set global cbo_prune_json_subfield = true;
set global query_mem_limit = 0;
set global enable_spill = true
r
It looks like you have a set of global configurations aimed at balancing performance and resource management. Here is a breakdown of what each of these settings does: Execution & Parallelism * 
enable_pipeline_engine = true
: Enables the Pipeline execution engine (the default in modern StarRocks versions), which improves CPU utilization and concurrency. * 
pipeline_dop = 0
: Sets the Degree of Parallelism (DOP) for the pipeline engine. 
0
means StarRocks will automatically determine the parallelism based on the number of CPU cores (usually half of the available cores). * 
enable_runtime_adaptive_dop = true
: Allows the system to dynamically adjust the parallelism at runtime based on the current system load. This helps prevent a single query from over-saturating the system while allowing it to scale up when resources are idle. * 
parallel_fragment_exec_instance_num = 1
: This is a legacy setting used before the pipeline engine. When 
enable_pipeline_engine
is 
true
, this is typically set to 
1
as 
pipeline_dop
takes over the parallelism control. Memory & Spilling * 
query_mem_limit = 0
: This sets the memory limit for a single query on each BE. 
0
means there is no strict limit per query (it will only be limited by the total BE process memory or resource group limits). * 
enable_spill = true
: Enables "Spill to Disk." If a query (like a large Join or Sort) exceeds the available memory, it will spill intermediate data to disk instead of failing with an "Out of Memory" (OOM) error. Note: This usually requires 
spill_mode
to be set (default is often 
auto
). Caching & Optimization * 
enable_query_cache = true
: Enables the Query Cache, which stores the results of identical SQL queries. This is very effective for high-concurrency dashboards where the same queries are executed frequently. * 
cbo_prune_json_subfield = true
: An optimization for JSON data types. It allows the Cost-Based Optimizer (CBO) to only read the specific subfields of a JSON column required by the query, significantly reducing I/O. Monitoring * 
enable_profile = false
: Disables the generation of detailed query profiles for successful queries. Turning this off reduces overhead and network traffic between the FE and BE, which is beneficial for high-concurrency environments. You should only set this to 
true
when debugging or tuning a specific slow query. Summary Table Variable | Value | Recommended Usage 
enable_query_cache
| 
true
| Good for repetitive queries/dashboards. 
enable_runtime_adaptive_dop
| 
true
| Good for mixed workloads. 
enable_spill
| 
true
| Prevents OOM for large queries at the cost of speed. 
enable_profile
| 
false
| Good for production to save overhead. Rocky's Tip: If you are using StarRocks v3.0+, these settings are generally quite solid for a production environment. However, ensure your BEs have enough disk space for spilling if you expect to process very large datasets that exceed your RAM. References • sql-reference/System_variable.md
b
@Rocky Anything else you recommend setting based on what I have?
r
To give you the most specific recommendations, it helps to know a bit more about your environment (like your StarRocks version, hardware specs, and whether you are doing heavy updates or just append-only analytics). However, based on common production setups, here are the top "must-have" settings and best practices I recommend checking: 1. Memory & Resource Management If you haven't tuned these, the defaults might either underutilize your hardware or lead to OOMs under heavy load: * 
mem_limit
(BE Config): Usually defaults to 90% of total RAM. Ensure this is appropriate for your machine. * 
query_mem_limit
(Session Variable): If you have large queries failing with "Memory limit exceeded," you can increase this globally or per session:
Copy code
sql
    SET GLOBAL query_mem_limit = 34359738368; -- example for 32GB
* `load_process_max_memory_limit_percent`: If you do heavy data loading, ensure this is set (default is 30%) to prevent loading from starving query resources. 2. Table Design (The "Big Three") Proper schema design usually yields more performance than any config parameter: * Sorting Keys: Ensure your most frequently used 
WHERE
clause columns are at the beginning of your 
DUPLICATE KEY
or 
PRIMARY KEY
definition. This enables prefix index scanning (extremely fast). * Bucketing (Distribution): Don't use too many buckets. A good rule of thumb is roughly 1GB to 10GB of data per tablet per partition. Too many small tablets hurt performance. * Partitioning: Use 
PARTITION BY RANGE
on time columns if you have a lot of data. This allows for partition pruning. 3. Query Tuning & Features * Enable Global Dictionary: StarRocks can automatically optimize 
COUNT(DISTINCT string_column)
using a global dictionary. Check if 
enable_low_cardinality_optimize
is 
true
. * Pipeline Engine: In newer versions (2.5+), the Pipeline engine is enabled by default. Ensure 
pipeline_dop
is set to 
0
(auto) so StarRocks can scale parallelism based on your CPU cores. * Resource Groups: If you have mixed workloads (e.g., automated reports vs. ad-hoc data science), set up Resource Groups to prevent one heavy user from slowing down the whole cluster. 4. Background Maintenance (Compaction) If you are using Primary Key tables with high-frequency updates: * 
compact_threads
(BE Config): In production, it is often recommended to set this to about 25% of your CPU cores to ensure background data merging stays ahead of incoming writes. * Monitor Compaction Score: Run 
SHOW PROC '/compactions';
to see if your "Score" is high (above 100 usually indicates a bottleneck). 5. Loading Optimization * Batch Size: Avoid "singleton inserts" (inserting one row at a time). Use Stream Load or Routine Load and aim for batches of at least 100MB or 100,000+ rows. * `enable_insert_strict`: Keep this 
true
(default) to ensure data quality by filtering out invalid rows during load. Would you like me to look into a specific area (like memory errors, slow queries, or loading speed)? Just let me know your StarRocks version and what you're seeing! References • loading/loading_introduction/loading_considerations.md • best_practices/overview.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • best_practices/table_clustering.md • best_practices/query_tuning/schema_tuning.md
b
@Rocky How do I see what the compaction score is?
r
You can view the compaction score for your tables and partitions in StarRocks using several methods. The Compaction Score reflects the merging status of data files; a higher score indicates more unmerged data versions. 1. Using 
SHOW PARTITIONS
 This is the most common way to check the compaction status for a specific table.
Copy code
sql
SHOW PARTITIONS FROM <table_name>;
In the output, look for the following columns: * MaxCS: The maximum compaction score among all tablets in the partition. * AvgCS: The average compaction score. * P50CS: The 50th percentile (median) compaction score. 2. Using 
information_schema.partitions_meta
 You can query the system-defined view to see compaction scores across multiple tables or the entire database.
Copy code
sql
SELECT 
    DB_NAME, 
    TABLE_NAME, 
    PARTITION_NAME, 
    MAX_CS, 
    AVG_CS 
FROM information_schema.partitions_meta 
ORDER BY MAX_CS DESC 
LIMIT 10;
3. Using 
SHOW PROC
 You can also navigate the internal process directory to see detailed partition information:
Copy code
sql
SHOW PROC '/dbs/<database_name>/<table_name>/partitions';
────────── What do the scores mean? * Below 10: Compaction is considered healthy or complete. * Above 100: The compaction score is high, and the system may begin to slow down data loading commits to let compaction catch up (
lake_ingest_slowdown_threshold
). * Above 500: The score is very high and might require manual intervention or an investigation into resource bottlenecks. * Above 2000: The system may reject new import transactions for that partition to prevent performance degradation (
lake_compaction_score_upper_bound
). Note: These specific thresholds for slowing down or rejecting imports primarily apply to Shared-data clusters. References • administration/management/compaction.md
b
@Rocky How much data cache memory is too much if my node is 120Gi RAM in size for a compute node. I set it to 40% or around 36GB
r
For a Compute Node (CN) with 120Gi RAM, setting 
datacache_mem_size
to 40% (~48Gi) is on the higher side but can be acceptable depending on your workload. However, for most StarRocks deployments, this might be considered "too much" memory for data caching at the expense of query execution space. Here is a breakdown of how to evaluate this setting: 1. Understanding Memory Allocation In a StarRocks CN/BE, memory is primarily split between: * Query Execution: Used for joins, aggregations, and sorting. This is governed by the 
mem_limit
parameter (default is 90% of total RAM, or ~108Gi in your case). * Data Cache (Memory): Controlled by 
datacache_mem_size
. This acts as a "Page Cache" in memory for frequently accessed data blocks. * Metadata & Overhead: A small portion for managing connections, metadata, etc. If you set 
datacache_mem_size
to 40% (48Gi) and your 
mem_limit
is 108Gi, you are leaving approximately 60Gi for query execution. 2. Is 40% Too Much? It is likely "too much" if: * You experience "Memory Limit Exceeded" (MLE) errors: If your queries are complex (large joins, heavy aggregations) and fail because they run out of memory, you are starving the execution engine by giving too much to the cache. * You have fast local SSDs/NVMe: In shared-data or data lake scenarios, StarRocks is highly optimized to use Disk Cache (
datacache_disk_size
). If you have fast disks, the performance difference between a 10% memory cache and a 40% memory cache is often negligible, but the risk to query stability is higher. 3. Recommended Best Practice For a 120Gi node, the typical recommendation is: * `datacache_mem_size`: Set to 10% to 20% (12Gi - 24Gi). This provides a fast "hot" tier for the most frequent data while leaving 80Gi+ for query execution. * `datacache_disk_size`: Rely on this for the bulk of your caching. Set this to use 80-90% of your local SSD space. StarRocks will automatically manage the lifecycle between disk and memory. 4. How to Monitor You can check if your current 40% setting is actually being utilized effectively by looking at the Query Profile: * DataCacheReadBytes: If this is high and your 
DataCacheWriteBytes
is low, your cache is working. * Query Execution Memory: If you see queries consistently hitting the 60Gi limit (remaining memory), you should decrease the cache percentage. Rocky's Tip: Unless your workload consists of very small, highly repetitive point lookups that fit entirely in 48Gi, I recommend dropping the memory cache to 20% (24Gi) and ensuring your local disk cache is fully utilized instead. This provides a much better balance for high-concurrency or complex analytical queries. You can adjust this dynamically without a restart to test:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("datacache_mem_size" = "20%");
-- Or via update be_configs if you want to target specific nodes
UPDATE be_configs SET VALUE="20%" WHERE NAME="datacache_mem_size";
(Note: Always update the CN configuration file to make the change persistent across restarts.) References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md • data_source/data_cache.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md
b
@Rocky How many tables and materialized views (if there is a difference) is too much in starrocks
r
In StarRocks, there is no hard "fixed" limit on the number of tables or materialized views (MVs) you can create. However, the practical limit is determined by your Frontend (FE) metadata capacity and total tablet count. Here is a breakdown of how to determine what is "too much" for your cluster: 1. The Real Bottleneck: Total Tablet Count The FE stores all metadata in its JVM heap memory. The more tables, partitions, and buckets you have, the more memory the FE requires. The general rule of thumb for FE memory sizing is based on the total number of tablets (tablets = partitions × buckets): Number of Tablets | Recommended FE RAM (-Xmx) < 1 million | 16 GB 1 – 2 million | 32 GB 2 – 5 million | 64 GB 5 – 10 million | 128 GB If you have 10,000 tables but each has only 1 tablet, you are fine. But if you have 1,000 tables and each has 1,000 tablets, your FE may experience high GC (Garbage Collection) pauses, slow startup times, or OOM (Out of Memory) crashes. 2. Tables vs. Materialized Views While both consume metadata, they impact the system differently: * Synchronous MVs: These are updated "on the fly" during data ingestion. Having too many (e.g., >5–10 per table) significantly slows down your data load speed because the system must write to the base table and all MVs simultaneously. * Asynchronous MVs: These behave more like independent tables. The main "cost" here is the background refresh resource (CPU/Memory on BEs) and the FE metadata overhead. * Nesting Limit: It is recommended not to exceed 3 levels of nested MVs (an MV built on another MV). * Rewrite Cache: If you have over 1,000 MVs used for transparent rewrite, you may need to increase the FE configuration 
mv_plan_cache_max_size
. 3. Signs You Have "Too Much" You should consider consolidating your schema if you notice: * Slow Metadata Operations: 
SHOW TABLES
, 
ALTER TABLE
, or 
CREATE TABLE
take several seconds. * FE Startup Time: If the FE takes a very long time to load the 
image
and 
edit_log
files. * Frequent Full GC: The FE Leader frequently restarts or steps down due to JVM pauses. * Small Files: Thousands of tables with very little data lead to "small file" problems on the storage layer, hurting query performance. Best Practices 1. Avoid "Over-partitioning": Don't create daily partitions if your data volume per day is only a few MBs. This inflates the tablet count unnecessarily. 2. Use Random Bucketing: For small tables, use 
DISTRIBUTED BY HASH(...) BUCKETS 1
or let StarRocks set the bucket count automatically to keep the tablet count low. 3. Cleanup: Regularly drop unused tables and old MVs. Every object in the system adds to the metadata lock contention. 4. Monitor: Watch the 
Tablets
metric in your monitoring dashboard (Grafana). If you are approaching 5-10 million tablets, it's time to either scale up your FE RAM or optimize your table designs. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/Others.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • data_source/catalog/iceberg/iceberg_catalog.md • administration/Meta_recovery.md
✅ 1
d
👀 was just going to ask these questions
💯 2
Open in Slack
PreviousNext
Powered by