<@U0A71G31CDV> If my table data is around ~10TB fo...
# questions-and-troubleshootingm
Minn Fong
06/04/2026, 1:55 PM@Rocky If my table data is around ~10TB for 24 hours worth, and I only keep 24 live hourly partitions in my primary key table, does it stand to reason that switching from shared nothing to shared data is recommended? The primary key size is on average 46 bytes for 5855128436 records. So a little more than 250GB needed to hold the PK index in memory. Given that each i4i.4xlarge machines have 128 GiB of RAM, then having 6 nodes is plenty right?
My understanding is each i4i.4xlarge has 3750 GiB of NVMe, so there will be 22.5 TB of NVMe cache space in total. Given this is double the size of the data, then it doesn't matter what query patterns happen to my cluster? Cache thrashing is a non-issue since the entire dataset can always live in the NVMe?
r
Rocky
06/04/2026, 1:56 PMBased on your data volume and table design, switching to a Shared-data (Cloud-Native) architecture is highly recommended. Here is a breakdown of why, along with specific considerations for your hardware and index management.
1. Is Shared-Data Recommended?
Yes. For a table with 10TB of daily data and a 24-hour rolling window, Shared-data offers several advantages:
* Elasticity: You can scale nodes up or down to handle peak query periods without the massive overhead of rebalancing 10TB of data across the network.
* Storage Efficiency: In Shared-nothing, you likely maintain 3 replicas (30TB of NVMe). In Shared-data, you store one primary copy in object storage (S3/GCS) and use your local NVMe as a cache, significantly reducing storage costs.
* Primary Key Management: Shared-data (since v3.1.0) fully supports Primary Key tables and, more importantly, supports Cloud-native Persistent Indexes (v3.3.2+), which are ideal for your scale.
2. Is 6 Nodes of
i4i.4xlarge(PK_size + 9) _ row_count _ 1.5(46 + 9) _ 5.85B _ 1.5enable_persistent_indextruedatacache.enable = trueDataCache"enable_persistent_index" = "true""persistent_index_type" = "CLOUD_NATIVE"i4im
Minn Fong
06/04/2026, 1:59 PM@Rocky What if I want to eventually increase the volume of my data retention to 4 days? Then not everything will fit in NVMe. Since there are no defined or controlled query patterns, random users can start querying oldest data and cause cache thrashing?
r
Rocky
06/04/2026, 1:59 PMIf you increase your data retention to 4 days (approx. 40TB), your working set will exceed your 22.5TB NVMe cache. However, StarRocks provides several mechanisms to prevent "random users" from causing cache thrashing and to ensure your hottest 24 hours remain performant.
1. Protection via SLRU Eviction
StarRocks' Data Cache uses a Segmented LRU (SLRU) strategy by default. Unlike a standard LRU, SLRU divides the cache into "eviction" and "protection" segments.
* How it prevents thrashing: When a user queries cold data (e.g., from Day 4), that data is placed in the eviction segment. It only moves to the protection segment if it is accessed multiple times.
* The Result: A one-time large scan of historical data will not immediately evict your frequently accessed "hot" data (Day 1).
2. Controlling the "Hot" Window
You can explicitly tell StarRocks which data is worth caching using the table property
datacache.partition_duration1 day Copy code
sql
-- Prevents this query from loading new data into the NVMe cache
SET enable_populate_datacache = false;
-- (Optional) Prevents the query from even checking the cache
SET enable_scan_datacache = false;
SELECT ... FROM my_table WHERE event_time < '2023-01-01';"persistent_index_type" = "CLOUD_NATIVE"l0_max_mem_usageCLOUD_NATIVEdatacache.partition_duration = "1 day"SET enable_populate_datacache = falsem
Minn Fong
06/04/2026, 2:04 PM@Rocky Is there any objective/subjecitve advantage to using shared nothign then
r
Rocky
06/04/2026, 2:04 PMWhile Shared-data is generally the superior choice for cloud-native elasticity and cost-efficiency (especially at your 10TB-40TB scale), Shared-nothing still offers specific objective and subjective advantages that might be relevant depending on your priorities.
1. Objective Technical Advantages
* Performance Determinism: In Shared-nothing, 100% of the data is local 100% of the time. You never deal with "cold start" latencies, cache warming tasks, or the variability of object storage (S3/GCS) performance.
* Multi-Replica Concurrency: Shared-nothing stores multiple replicas of the data (typically 3). This allows StarRocks to load-balance queries across different nodes holding the same data. In Shared-data, the cache is a single copy; if a node is busy or fails, the next node must pull that data from the remote bucket.
* High Availability for Compute: Because data is replicated locally, if a node goes down, the cluster can immediately route queries to another node with a local copy. In Shared-data, a new node (or an existing one) must first download/cache the data from the network before it can match the performance of the failed node.
* Feature Completeness: While the gap is closing rapidly, some advanced optimizations (like Sorted Streaming Aggregate) are currently only supported in Shared-nothing.
* Zero External Dependencies: Shared-nothing does not require an S3-compatible object store or a high-bandwidth connection to one. This is ideal for on-premises, air-gapped, or edge environments where local NVMe is the only reliable resource.
2. Subjective & Operational Advantages
* Architectural Maturity: Shared-nothing is the original foundation of StarRocks. For extremely conservative organizations, it is often viewed as the "tried and true" path with fewer variables to tune (no cache eviction policies, starlet configurations, etc.).
* Simplified Troubleshooting: When a query is slow in Shared-nothing, you look at CPU, Memory, and Disk I/O. In Shared-data, you have to add "Object Storage Latency," "Network Bandwidth," and "Cache Hit Rate" to your root cause analysis.
* "What You See Is What You Get": You don't have to worry about a "random user" causing cache thrashing (as you mentioned). The performance profile of the cluster remains static regardless of whether the user is querying data from 1 hour ago or 1 week ago (provided it fits on the disks).
Is it right for your 10TB-40TB use case?
For your specific scenario (10TB daily, potentially 40TB total), Shared-nothing would require massive local storage (40TB x 3 replicas = 120TB of NVMe).
* Shared-nothing would give you more "peace of mind" regarding performance stability, but at a 3x-5x higher storage cost.
* Shared-data is the industry standard for your volume because it allows you to pay for 40TB of "cheap" S3 storage while keeping only the 10TB of "active" data in your expensive NVMe cache.
Conclusion: Unless you have a strict requirement for "zero-variability" performance or are running in a restricted environment without object storage, the Shared-data advantages of cost and elasticity far outweigh the Shared-nothing benefits for a 40TB dataset.
References
• using_starrocks/sorted_aggregate.md
• table_design/table_types/unique_key_table.md
• best_practices/query_tuning/schema_tuning.md
• table_design/table_types/table_capabilities.md
m
Minn Fong
06/04/2026, 3:10 PM@Rocky Can I re-use my shared nothing cluster FE nodes to also direct traffic to my new shared data cluster? Or each architecture needs it's own FE node?
Also, what is the process like for bringing up a CN node if it goes down and refreshing cache from S3? I know for BE node it is simple, just reattach the EBS with script
r
Rocky
06/04/2026, 3:11 PM1. Can you re-use FE nodes for both architectures?
No. In StarRocks, the
run_modefe.confshared_nothingshared_datarun_mode = shared_datastarlet_cachedatacache Copy code
sql
-- Proactively load the last 24 hours into NVMe
CACHE SELECT * FROM my_table WHERE event_time >= '2023-10-27 00:00:00';m
Minn Fong
06/04/2026, 3:12 PM@Rocky But if a CN node goes down, it uses NVMe instance store, means data is gone?
r
Rocky
06/04/2026, 3:12 PMYes, if you use NVMe instance stores (which are ephemeral), the cache data on that specific node is indeed gone when the instance is stopped or terminated. However, it is important to distinguish between data loss and cache loss.
1. Your Table Data is Safe (Source of Truth)
In a Shared-data architecture, the Source of Truth is S3 (Object Storage).
* When you "write" data to StarRocks, it is committed to S3 before the transaction is considered successful.
* The NVMe instance store only holds a copy of that data for speed.
* If the CN node vanishes, 0% of your actual table data is lost. You could delete every single CN node in your cluster, and your data would still be perfectly safe in S3.
2. What happens to the "Gone" Data?
When a new CN node comes up (or the old one restarts with a fresh, empty NVMe):
* Cold Start Performance: The first time a user queries "Day 1" data, the CN sees that its local NVMe is empty (Cache Miss). It will automatically fetch the data from S3.
* Automatic Re-population: As it fetches data from S3 to answer the query, it simultaneously writes that data back onto the new NVMe. The second time that same data is queried, it will be a "Cache Hit" and run at full NVMe speed.
3. What about the Primary Key Index?
This is the only area where "data being gone" from the disk matters for recovery time.
* Standard Persistent Index: If stored only on the ephemeral NVMe, the CN would have to rebuild the index by downloading metadata from S3 when it starts. For a 23 billion row table, this could take some time.
* The Solution (Cloud-Native Persistent Index): StarRocks supports a
CLOUD_NATIVECLOUD_NATIVEm
Minn Fong
06/04/2026, 3:23 PM@Rocky Does configuring the most recent "n" partitions to only be allowed in NVMe cache mean I solve cache thrashing issues with rogue queries?
r
Rocky
06/04/2026, 3:23 PMYes, configuring
datacache.partition_durationstorage_cache_ttldatacache.partition_durationpartition_duration30 DAY Copy code
sql
CREATE TABLE my_table ( ... )
PROPERTIES (
"datacache.enable" = "true",
"datacache.partition_duration" = "7 DAY" -- Only cache the last 7 days of data
);partition_duration Copy code
sql
SET skip_local_disk_cache = true; -- (v3.3.9+) Bypasses NVMe cache
SELECT ... ;partition_duration3 DAY7 DAYm
Minn Fong
06/04/2026, 3:33 PM@Rocky How about compaction? How does it worked on shared data? Is it less of an issue in shared data architecture? Because I had to do a lot of cluster tuning to get compaction down for my shared nothing cluster. For instance, I needed to reduce tablet compaction interval to 10 seconds, and 12 compact threads. This all just to keep compaction score below 250. I already do large batching of my data (500MB) with stream load enabled.
r
Rocky
06/04/2026, 3:34 PMCompaction in a Shared-data (Cloud-native) architecture is managed very differently than in Shared-nothing. While it is still a critical background process, it is generally easier to manage because of how resources are decoupled.
1. The Key Difference: FE-led Scheduling
In Shared-nothing, each BE is responsible for its own local tablets. In Shared-data, the FE acts as the "brain" for compaction across the entire cluster:
* Centralized Scores: The Leader FE tracks the "Compaction Score" for every partition in the cluster.
* Dynamic Dispatch: The FE picks the "hottest" partitions (highest scores) and dispatches them as tasks to any available CN node.
* Decoupled IO: Because data lives in S3, any CN can compact any tablet. You no longer have the "hot node" problem where one BE is stuck with 500 tasks while others are idle.
2. Is it "less of an issue"?
Yes and No.
* Why it's easier: If your compaction score is climbing, you can simply scale out your CN nodes. Since compaction isn't tied to where the data is stored (it's all in S3), adding 5 more CNs immediately increases the cluster's "compaction throughput." In Shared-nothing, adding a node requires a long rebalancing process before that node can help with existing data.
* The Trade-off: Compaction now involves S3. A CN must download data from S3 (if not in cache), merge it, and upload the result back to S3. This consumes network bandwidth and S3 PUT/GET requests, though StarRocks optimizes this by using the local NVMe cache during the merge process.
3. Tuning for Your Scale (High Throughput)
Since you are already doing 500MB batches (which is excellent), you likely won't need the "10-second interval" hacks you used in Shared-nothing. Instead, you focus on concurrency.
FE Tuning (Global)
*
lake_compaction_max_tasks-1CN_nodes * 16lake_ingest_slowdown_threshold100lake_compaction_score_selector_min_score10.0compact_threadsmax_cumulative_compaction_num_singleton_deltas Copy code
sql
-- View global scores (Look for MaxCS)
SELECT * FROM information_schema.partitions_meta ORDER BY MAX_CS DESC;
-- View active tasks and their progress (%)
SELECT * FROM information_schema.be_cloud_native_compactions;
-- View history of compaction tasks
SHOW PROC '/compactions';compact_threadsmax_cumulative_compaction_num_singleton_deltas = 100Rocky
06/04/2026, 3:34 PMm
Minn Fong
06/04/2026, 3:36 PM@Rocky Given the configurations I needed for my shared nothing architecture of such frequent compactions, that means shared data will be even more punishing? It has to fetch merge and upload back compacted data to S3 so frequently.
r
Rocky
06/04/2026, 3:36 PMIt is a common concern that the "S3 round-trip" (fetch-merge-upload) would be slower, but in practice, the Shared-data architecture is actually less punishing for high-ingestion workloads.
Here is why your "Shared-nothing" tuning logic changes and why the overhead is manageable:
1. You no longer have "Hot Node" bottlenecks
In your Shared-nothing cluster, if one node received more data, it had to handle all the compaction for those tablets using its local CPU/Disk. If it fell behind, your whole cluster's ingestion would slow down.
* Shared-data Reality: Compaction is stateless. If Partition A has a high score, the FE can send that task to any idle CN node. The work is distributed globally across the cluster, preventing any single node from being "punished" into a corner.
2. Compaction is "Cache-Aware"
The "Fetch" part of the cycle isn't as heavy as it sounds:
* Local NVMe as a Buffer: When a CN node performs compaction, it reads data. If that data is already in the local NVMe cache (which it likely is, since you just wrote it), the "Fetch" is local-speed, not S3-speed.
* Vertical Compaction: StarRocks uses "Vertical Compaction" to merge data column-by-column rather than row-by-row. This significantly reduces the memory footprint and I/O required during the merge.
* Filling the Cache: There is a setting (
lake_enable_vertical_compaction_fill_data_cachelake_compaction_max_tasksRocky
06/04/2026, 3:36 PM• knowledge/admin_and_operation/shared_data_compaction.md
• faq/shared_data_faq.md
• deployment/shared_data/feature-support-shared-data.md
• using_starrocks/caching/block_cache.md
• administration/management/BE_configuration.md
• administration/management/FE_configuration.md
• sql-reference/System_variable.md
• administration/management/compaction.md
• knowledge/admin_and_operation/shared_data_compaction.md
• faq/shared_data_faq.md
• deployment/shared_data/feature-support-shared-data.md
• using_starrocks/caching/block_cache.md
• administration/management/BE_configuration.md
• administration/management/FE_configuration.md
• sql-reference/System_variable.md
References
• knowledge/admin_and_operation/shared_data_compaction.md
• administration/management/compaction.md
• administration/management/FE_configuration.md
• faq/shared_data_faq.md
m
Minn Fong
06/05/2026, 4:30 PM@Rocky If a CN node goes down, the process of making it operational again is much easier than if a BE node goes down in shared nothing right? Because for BE we have to remount the EBS volumes. The CN node just needs to come back up and queries routed to it will automatically replenish it's NVMe cache?
r
Rocky
06/05/2026, 4:30 PMYes, you are exactly right. The operational overhead for recovering a CN node in a Shared-data architecture is significantly lower than a BE node in Shared-nothing.
Here is the breakdown of why the CN recovery process is "easier":
1. No Volume Management (EBS)
* Shared-nothing (BE): Since each BE owns a unique subset of the data, you are tied to the specific EBS volumes where those tablets live. If a BE instance fails, you must either remount those specific EBS volumes to a new instance (which can be slow or error-prone) or wait for the cluster to re-replicate that data from other BEs (which causes massive network/IO pressure).
* Shared-data (CN): The NVMe disk on a CN is strictly for cache. It does not hold the "source of truth." If a CN node goes down, you can simply spin up a brand new instance with a fresh, empty NVMe disk. There is no need to worry about attaching old volumes or recovering "lost" data because the data is already safe in S3.
2. Instant Operational Readiness
* Shared-nothing (BE): When a BE comes back, it must report thousands of tablet versions to the FE and potentially undergo a "consistency check."
* Shared-data (CN): A CN node is essentially stateless. Once the process starts and registers with the FE, it is immediately ready to take queries. The FE knows that all data is in S3, so it doesn't need the CN to "prove" what data it has.
3. Cache Replenishment (Reactive vs. Proactive)
As you suspected, the cache is replenished automatically, but you have options:
* Reactive (Default): When queries are routed to the new CN, it will see a cache miss, fetch the data from S3, and write it to the local NVMe. Performance starts "cold" and gets "hot" as users run their dashboards.
* Proactive (Manual/Scheduled): If you want to avoid that "cold start" period for your most important tables, StarRocks (v3.3.0+) provides a
CACHE SELECT Copy code
sql
-- Warm up the last 3 days of data for a specific table
CACHE SELECT * FROM sales_table WHERE dt >= '2024-01-01';m
Minn Fong
06/05/2026, 5:15 PM@Rocky What is the config name I should use or table property to specify such that only the most recent "n" partitions of data are held in data cache (NVMe)
r
Rocky
06/05/2026, 5:15 PMFor a Shared-data (cloud-native) table, you achieve this using the
datacache.partition_durationdatacache.partition_durationstorage_cache_ttlYEARMONTHDAYHOUR Copy code
sql
CREATE TABLE orders (
order_id BIGINT,
order_date DATE,
...
)
PARTITION BY RANGE(order_date) (...)
PROPERTIES (
"datacache.enable" = "true",
"datacache.partition_duration" = "7 DAY"
);partition_duration Copy code
sql
ALTER TABLE orders SET ("datacache.partition_duration" = "14 DAY");partition_ttl_numberpartition_durationm
Minn Fong
06/05/2026, 5:20 PM@Rocky What are the benefits of shared data over shared nothing in the context of high throughput primary key table insertion and partial updates? Recommend pros and cons for each architecture and your recommendation. Also, what family of ec2 instances would be ideal? My current inclination is the i4i machines since they have a lot of NVMe.
r
Rocky
06/05/2026, 5:20 PMIn the context of high-throughput Primary Key (PK) tables with frequent partial updates, the Shared-data architecture is generally the superior choice for modern workloads. While Shared-nothing offers lower raw storage latency, the operational and stability benefits of Shared-data for high-concurrency "upserts" outweigh the minor S3 overhead.
Recommendation: Shared-data (Cloud-native)
For a workload involving frequent partial updates and high throughput, Shared-data is recommended because it transforms "compaction" from a local bottleneck into a globally distributable task.
──────────
Architecture Comparison (PK Tables & Partial Updates)
Feature | Shared-nothing (BE) | Shared-data (CN)
Write Stability | High risk of "Hot Nodes." If one node gets more updates, its compaction scores spike and block ingestion. | Distributed Compaction. FE assigns compaction tasks to any available CN node, balancing the load cluster-wide.
Partial Updates | Supports Row and Column mode. High IOPS on local disks during merges. | Supports both (Column mode requires v3.3.1+). Uses local NVMe cache for merging to match local speeds.
Scaling | Scaling out requires Data Rebalancing, which is IO-heavy and risky during high-throughput ingestion. | Instant Scaling. Add CN nodes to immediately increase compaction throughput with zero data movement.
PK Index | Persistent index stored on local disk. Tied to specific BE instance. | Cloud-native Persistent Index (v3.3.2+). Index data is on S3 with local caching. No re-indexing needed if nodes fail.
Cost | Fixed cost of EBS volumes. Pay for provisioned IOPS. | Pay-as-you-go S3 storage. S3 API costs can rise with very high-frequency small writes.
──────────
Why Shared-data wins for your use case
1. Global Compaction Scheduling: In Shared-nothing, if Table A's updates land mostly on BE-1, BE-1's CPU/IO will be consumed by compaction while BE-2 sits idle. In Shared-data, the FE sees the "Compaction Score" for a partition and can send the task to any CN. This prevents the "Compaction Death Spiral" where one node falling behind slows down the whole cluster.
2. Column-Mode Partial Updates: StarRocks v3.3+ significantly optimized partial updates in Shared-data. By using the Column Mode, only the modified columns are rewritten, which reduces the amount of data fetched from and uploaded back to S3.
3. Cloud-Native Persistent Index: This is a game-changer for large PK tables. It offloads the memory-heavy PK index to S3 and caches only the "hot" parts of the index on your NVMe. This allows you to handle billions of rows without needing massive RAM on every node.
──────────
Instance Recommendation:
i4ii4ii4ii4ii4im6idc6idm6ir6i Copy code
sql
PROPERTIES (
"enable_persistent_index" = "true",
"persistent_index_type" = "CLOUD_NATIVE"
);Rocky
06/05/2026, 5:20 PMThis ensures that if you cycle your
i4im
Minn Fong
06/05/2026, 5:23 PM@Rocky But isn't the issue with column mode that it is ideal for a small set of columns to update? If I have 9 different partial update schemas, the Delta Column Group is big, and row mode can be more efficient? I observed this trend in my shared nothing cluster where the column partial update duration was higher than update rowset commit apply duration
r
Rocky
06/05/2026, 5:23 PMYou are absolutely correct in your observation. When you have 9 different partial update schemas, the overhead of managing Delta Column Groups (DCGs) in Column Mode can exceed the benefits, often making Row Mode the better choice for high-throughput Primary Key (PK) tables.
1. Row Mode vs. Column Mode (The "9 Schema" Problem)
The performance gap you observed in your Shared-nothing cluster regarding "update rowset commit apply duration" is likely due to how StarRocks handles updates:
* Column Mode (Read-Optimized): This mode creates Delta Column Groups. If you have 9 different update schemas, StarRocks has to track 9 different sets of updated columns. During the "Apply" phase (commit), the system must perform complex merging of these delta groups. If the number of columns updated is high or the schemas are diverse, the CPU and IO cost to "Apply" these changes into the columnar storage becomes a bottleneck.
* Row Mode (Write-Optimized): Row mode stores the updated rows in a hidden binary
__row Copy code
sql
PROPERTIES (
"partial_update_mode" = "row"
);i4ii4ii4i Copy code
sql
CREATE TABLE my_table (
pk_id BIGINT,
...
) PRIMARY KEY (pk_id)
DISTRIBUTED BY HASH(pk_id)
PROPERTIES (
"enable_persistent_index" = "true",
"persistent_index_type" = "CLOUD_NATIVE",
"partial_update_mode" = "row",
"datacache.enable" = "true"
);Rocky
06/05/2026, 5:23 PM_ _Pro:* Operational simplicity; handles high-concurrency compaction; near-instant node recovery.
_ _Con:* Slightly higher S3 API costs for very small, frequent commits (use "Transaction Batching" to mitigate this).
References
• loading/Load_to_Primary_Key_tables.md
• table_design/table_types/primary_key_table.md
• sql-reference/sql-statements/table_bucket_part_index/UPDATE.md
• deployment/plan_cluster.md
References
• loading/Load_to_Primary_Key_tables.md
• table_design/table_types/primary_key_table.md
• sql-reference/sql-statements/table_bucket_part_index/UPDATE.md
• table_design/hybrid_table.md
• administration/management/BE_configuration.md
• knowledge/admin_and_operation/shared_data_compaction.md
• deployment/plan_cluster.md
• loading/Spark-connector-starrocks.md
• knowledge/usage/best-practice.md
• table_design/table_types/table_capabilities.md
• administration/management/monitoring/metrics.md
References
• sql-reference/sql-statements/table_bucket_part_index/UPDATE.md
• knowledge/usage/best-practice.md
• administration/management/BE_configuration.md
• knowledge/admin_and_operation/shared_data_compaction.md
m
Minn Fong
06/05/2026, 5:28 PM@Rocky Can I use my transaction per second metric in my shared nothing cluster as an indicator of the number of S3 PUT operations I would make in a shared data cluster?
r
Rocky
06/05/2026, 5:28 PMNo, your Transactions Per Second (TPS) metric is not a 1:1 indicator of S3 PUT operations. In a Shared-data (cloud-native) cluster, a single logical transaction often results in multiple physical S3 PUT operations.
If you are planning your S3 API budget or performance, you need to account for the following "amplification" factors:
1. The Tablet Multiplier
When you commit a transaction, StarRocks writes data files (segments) to S3.
* Formula:
PUTs per Transaction ≈ Number of Tablets TouchedCLOUD_NATIVEi4ilake_enable_batch_publish_version(TPS _ Avg_Tablets_Touched) _ 1.51.5fslib_s3_single_upload_iopsm
Minn Fong
06/05/2026, 6:56 PM@Rocky Is stream load my best option for data ingestion? I receive data via grpc stream in my services
r
Rocky
06/05/2026, 6:57 PMYes, Stream Load is your best option for ingesting data from a gRPC service. Because you already have a middle-tier service handling the gRPC stream, you can implement the "Batching" logic there, which is critical for the Shared-data architecture we discussed.
Here is how to optimize Stream Load for your gRPC stream, particularly for Primary Key tables on Shared-data:
1. Choice of Stream Load Method
Depending on your StarRocks version and your service's concurrency, you have three strategies:
* Strategy A: Manual In-Memory Batching (Simplest)
* How: Your service accumulates records from the gRPC stream into a buffer (e.g., 10MB or 10,000 rows). Once the buffer is full or a timer (e.g., 5s) expires, you trigger a standard HTTP Stream Load.
* Why: This is the most stable and allows you to control exactly how many S3 PUT operations you generate.
* Strategy B: Transaction Interface (v2.4+)
* How: Call
/api/transaction/beginlabel/api/transaction/load/api/transaction/commitenable_merge_commit Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");PROPERTIES ("function_column.sequence_col" = "your_timestamp_column")Rocky
06/05/2026, 6:57 PMm
Minn Fong
06/05/2026, 7:06 PM@Rocky Does compaction always read, compact, then write back to S3? What if my data always fit inside the NVMe? Does it make things faster?
r
Rocky
06/05/2026, 7:07 PMIn a Shared-Data (Lake) cluster, the short answer is: Yes, it always writes back to S3, but it doesn't always have to read from S3.
Here is the breakdown of how compaction interacts with your
i4ii4ilake_enable_vertical_compaction_fill_data_cache = truedatacache_read_bytesremote_read_bytesremote_read_bytesm
Minn Fong
06/08/2026, 1:42 PM@Rocky So if I have plenty of NVMe, more than enough to house my data, does this mean compaction never has to read from S3? Just compact locally and write?
r
Rocky
06/08/2026, 1:42 PMExactly. If your NVMe capacity exceeds your total data size, you have effectively turned your Shared-data cluster into a "Local-Read, Cloud-Write" system for compaction.
Here is exactly how that works and why it significantly boosts performance:
1. Zero-Read S3 Compaction
When compaction begins, the Compute Node (CN) needs to read multiple old segment files to merge them.
* The Workflow: The CN checks its Data Cache (NVMe) for these blocks. Since your data fits entirely in NVMe, the cache hit rate will be 100%.
* Result: The CN pulls all the "input" for compaction at NVMe speeds (GBs per second) with zero S3 API "GET" costs and zero network latency.
2. The "Fill Cache" Write-Back
Even though the CN must write the new, merged file to S3 (to ensure the cloud "Source of Truth" is updated), it doesn't wait to re-download it later.
* Feature:
lake_enable_vertical_compaction_fill_data_cachetrue${storage_root_path}/persist/m
Minn Fong
06/08/2026, 1:45 PM@Rocky Does datacache_read_bytes represent total cache bytes read, be it for queries or for compaction?
r
Rocky
06/08/2026, 1:45 PMIt depends on where you are seeing that specific metric name. StarRocks distinguishes between foreground (query) and background (compaction) I/O to help you monitor system health accurately.
1. If you see it in a Query Profile
In the context of a Query Profile,
DataCacheReadBytesDataCacheWriteBytesdatacache_read_bytesblock_cache_hit_bytesDataCacheReadBytescompaction_bytes_totalfslib_total_read_dataDataCacheReadBytesblock_cache_hit_bytesm
Minn Fong
06/08/2026, 2:21 PM@Rocky Two questions
1. Doesn't the CN nodes have to make additional S3 calls to delete the uncompacted files that were merged?
2. How does the CN cluster as a whole share the load of caching data in NVMe? Like 10 TB of data should be spread equally across all CN nodes? Does data in cache get determined by the distribution key? If that's the case, what happens when a CN node goes down. Does the distribution get modulo by the remaining CN nodes? So the remaining n-1 CN nodes have to split fetch the data that was on the CN node that went down?
r
Rocky
06/08/2026, 2:22 PMGreat questions! These get into the weeds of how StarRocks manages its cloud-native (Shared-Data) architecture.
1. Does the CN node make S3 calls to delete uncompacted files?
Yes, but it's a two-stage process.
* Stage 1: Versioning (The Commit): When a CN node finishes a compaction task, it writes the new files to S3 and tells the FE (Leader). The FE then "publishes" a new version of the metadata. At this exact moment, the old files are not deleted; they are simply no longer "visible" to new queries.
* Stage 2: Vacuum (The Deletion): To ensure queries already in progress can finish reading those old files, StarRocks waits for a "grace period" (controlled by the FE config
lake_autovacuum_grace_period_minutesDeleteObjectsDeleteObjectsorder_idTabletID % NRocky
06/08/2026, 2:22 PMm
Minn Fong
06/08/2026, 2:36 PM@Rocky What happens to the newly distributed tablets once the dead CN node comes back up and refreshes it's cache? DO they get evicted? Also, is there a cache sharing feature?
r
Rocky
06/08/2026, 2:37 PMWhen a dead CN node returns or you add a new node to the cluster, the Consistent Hashing mechanism ensures the cluster stabilizes with minimal data movement. Here is exactly what happens to your tablets and the cache:
1. Tablet Re-distribution: The Return Home
Because StarRocks uses consistent hashing (managed by a component called Starlet), each tablet has a "preferred" owner.
* Automatic Re-assignment: As soon as the FE detects the CN node is back online (via heartbeats), the hashing algorithm automatically maps the original tablets back to that node.
* The Switch: The "temporary" owner nodes stop being responsible for those tablets. New queries for that data are routed back to the recovered CN node.
2. What happens to the "Temporary" Cache?
The data that was cached on the $n-1$ nodes (while the primary node was down) does not get deleted immediately.
* Passive Eviction: The data remains on the temporary nodes' NVMe. Since those nodes are no longer the "owners" of those tablets, they won't receive new queries for that data. Over time, as those nodes cache their own new data, the old "inherited" data will naturally be evicted based on LRU (Least Recently Used) or SLRU logic.
* Disk Usage: This is why you might see NVMe disk usage stay high even after a node returns; StarRocks doesn't waste I/O "cleaning up" if the space isn't needed yet.
3. Cache Sharing: The "S3 Bypass" Feature
You asked if there is a cache sharing feature—*Yes, there is.* It was designed specifically to prevent the "S3 fetch storm" when a node is new or just recovered.
* Feature Name: Cache Sharing (introduced in v3.x).
* How it works: If a CN node needs data that it doesn't have in its local NVMe (a "cache miss"), it doesn't immediately go to S3. It first checks if any other CN node in the cluster has that block in its cache. If a peer has it, the node fetches the data over the local network (10Gbps/25Gbps) instead of S3.
* Why it matters:
* Speed: Network transfers between nodes are often faster and lower latency than S3.
* Cost: You avoid S3 "GET" request fees.
* Warm-up: It allows your recovered node to "re-warm" its cache from its neighbors rather than the cloud.
How to enable Cache Sharing:
You need to set both a global variable and an FE configuration:
1. FE Config:
ADMIN SET FRONTEND CONFIG ("enable_trace_historical_node" = "true");SET GLOBAL enable_datacache_sharing = true;CACHE SELECTRocky
06/08/2026, 2:37 PM• administration/management/FE_configuration.md
• faq/shared_data_faq.md
• introduction/Architecture.md
• sql-reference/sql-statements/data-definition/CREATE_TABLE.md
• sql-reference/System_variable.md
• table_design/Data_distribution.md
• table_design/table_types/Primary_key_table.md
References
• deployment/shared_data/feature-support-shared-data.md
• deployment/shared_data/minio.md
• deployment/shared_data/s3.md
• deployment/shared_data/azure.md
m
Minn Fong
06/08/2026, 2:46 PM@Rocky List important metrics I should monitor for a shared data cluster primary key table with high throughput ingestion (prometheus metric names)
r
Rocky
06/08/2026, 2:47 PMMonitoring a high-throughput Primary Key (PK) table in a shared-data cluster requires tracking ingestion stability, PK-specific memory overhead, and the health of the object storage (S3/OSS) interface.
1. Ingestion & Compaction Health
For PK tables, high throughput can lead to "version explosion" if compaction can't keep up.
*
starrocks_fe_tablet_max_compaction_scorestarrocks_be_lake_delta_writer_queue_sizestarrocks_be_lake_publish_version_latencystarrocks_be_mem_tracker{type="update"}starrocks_be_lake_pk_index_write_bytesstarrocks_be_lake_pk_index_read_bytesstarrocks_be_lake_delvec_cache_miss_per_minutestarrocks_be_lake_get_tablet_metadata_latencystarrocks_be_lake_metadata_cache_miss_per_minutestarrocks_be_block_cache_hit_bytesstarrocks_be_block_cache_lookups_totalstarrocks_be_fslib_read_throughputstarrocks_be_fslib_write_throughputstarrocks_be_fslib_read_io_latencystarrocks_be_lake_vacuum_delete_totalstarrocks_be_lake_vacuum_errors_totalstarrocks_fe_tablet_max_compaction_scorestarrocks_be_lake_publish_version_latencystarrocks_be_mem_tracker{type="update"}starrocks_be_block_cache_hit_ratiostarrocks_be_fslib_read_io_latencyRocky
06/08/2026, 2:47 PMm
Minn Fong
06/08/2026, 3:12 PM@Rocky many of these metrics are not showing up on grafana. For instance, starrocks_be_block_cache_hit_bytes, starrocks_be_fslib_rtead_throughput not showing up.
r
Rocky
06/08/2026, 3:13 PMIt is common for some "Shared-Data" or "Data Cache" metrics to be missing from Grafana if you are looking for the standard
starrocks_be_starrocks_be_starrocks_be_fslib_read_throughputfslib_read_throughputstarrocks_be_block_cache_hit_bytesstarrocks_be_datacache_read_block_countstarrocks_be_datacache_read_block_bytesdatacache/metricshttp://<BE_IP>:<BE_HTTP_PORT>/api/datacache/statenable_datacache = falsedatacache_block_cache_fslib_read_throughputcurl http://<BE_IP>:8040/metrics | grep block_cachestarrocks_be_lake_...starrocks_be_lake_publish_version_latencystarrocks_be_datacache_read_bytesstarrocks_be_datacache_hit_bytesstarrocks_be_fslib_read_io_latencystarrocks_be_lake_delta_writer_queue_sizem
Minn Fong
06/08/2026, 3:21 PM@Rocky These are the only cache_hit / fslib metrics shown
Copy code
starrocks_be_page_cache_hit_count 151896739439
# TYPE starrocks_be_query_cache_hit_count gauge
starrocks_be_query_cache_hit_count 0
# TYPE starrocks_be_query_cache_hit_ratio gauge
starrocks_be_query_cache_hit_ratio 0.000000
# TYPE starrocks_be_vector_index_cache_hit_count gauge
starrocks_be_vector_index_cache_hit_count 0
# TYPE starrocks_be_vector_index_cache_hit_ratio gauge
starrocks_be_vector_index_cache_hit_ratio 0.000000
# HELP default_cache_hit_bytes
# TYPE default_cache_hit_bytes gauge
default_cache_hit_bytes 0
# HELP default_cache_hit_bytes_last_minite
# TYPE default_cache_hit_bytes_last_minite gauge
default_cache_hit_bytes_last_minite 0
# HELP default_cache_hit_count
# TYPE default_cache_hit_count gauge
default_cache_hit_count 0
# HELP default_cache_hit_count_last_minite
# TYPE default_cache_hit_count_last_minite gauge
default_cache_hit_count_last_minite 0
# HELP lake_delvec_cache_hit_minute
# TYPE lake_delvec_cache_hit_minute gauge
lake_delvec_cache_hit_minute 0
# HELP lake_metadata_cache_hit_minute
# TYPE lake_metadata_cache_hit_minute gauge
lake_metadata_cache_hit_minute 0
# HELP lake_schema_cache_hit_minute
# TYPE lake_schema_cache_hit_minute gauge
lake_schema_cache_hit_minute 0
# HELP lake_segment_cache_hit_minute
# TYPE lake_segment_cache_hit_minute gauge
lake_segment_cache_hit_minute 0
# HELP lake_txn_log_cache_hit_minute
# TYPE lake_txn_log_cache_hit_minute gauge
lake_txn_log_cache_hit_minute 0
# HELP merge_commit_txn_cache_hit_total
# TYPE merge_commit_txn_cache_hit_total gauge
merge_commit_txn_cache_hit_total 0
100 241963 100 241963 0 0 46168# HELP table_schema_service_schema_cache_hit
k# TYPE table_schema_service_schema_cache_hit gauge
table_schema_service_schema_cache_hit 0
0# HELP fslib_open_cache_hits Count how many hits to cache file when opening a file for read.
# TYPE fslib_open_cache_hits counter
--fslib_open_cache_hits{fstype="cachefs"} 0
100 242075 100 242075 0 0 38747k # HELP fslib_file_replicate_count How many files are replicated.
# TYPE fslib_file_replicate_count counter
fslib_file_replicate_count 0
0# HELP fslib_file_replicate_fail_count How many files fail to replicate.
--:--:-# TYPE fslib_file_replicate_fail_count counter
-fslib_file_replicate_fail_count 0
# HELP fslib_fs_open_files Count how many files are opened by given filesystem
--# TYPE fslib_fs_open_files counter
:-fslib_fs_open_files{fstype="cachefs"} 0
-:# HELP fslib_fs_create_files Count how many files are created by given filesystem
--# TYPE fslib_fs_create_files counter
-fslib_fs_create_files{fstype="cachefs"} 0
-:--:--# HELP fslib_entry_stat Counter of fs stat calls
# TYPE fslib_entry_stat counter
3fslib_entry_stat{fstype="cachefs"} 0
9# HELP fslib_entry_exists Counter of fs exists calls
4# TYPE fslib_entry_exists counter
0fslib_entry_exists{fstype="cachefs"} 0
0k# HELP fslib_list_dir Counter of fs list directory calls
# TYPE fslib_list_dir counter
fslib_list_dir{fstype="cachefs"} 0
# HELP fslib_fs_delete_files Count how many files are deleted by given filesystem
# TYPE fslib_fs_delete_files counter
fslib_fs_delete_files{fstype="cachefs"} 0
# HELP fslib_open_cache_hits Count how many hits to cache file when opening a file for read.
# TYPE fslib_open_cache_hits counter
fslib_open_cache_hits{fstype="cachefs"} 0
# HELP fslib_open_cache_misses Count how many misses to cache file when opening a file for read.
# TYPE fslib_open_cache_misses counter
fslib_open_cache_misses{fstype="cachefs"} 0
# HELP fslib_star_cache_exist_calls Count how many times of the exist api has been called.
# TYPE fslib_star_cache_exist_calls counter
fslib_star_cache_exist_calls{fstype="cachefs"} 0
# HELP fslib_active_replicate_file_handle_count How many files are being replicated.
# TYPE fslib_active_replicate_file_handle_count gauge
fslib_active_replicate_file_handle_count 0
# HELP fslib_cache_threadpool_size Count CacheManager threadpool threads
# TYPE fslib_cache_threadpool_size gauge
fslib_cache_threadpool_size{fstype="cachefs"} 16
# HELP fslib_async_file_replicate_wait_count How many file are waiting to be replicated.
# TYPE fslib_async_file_replicate_wait_count gauge
fslib_async_file_replicate_wait_count 0
# HELP fslib_async_file_wait_timeout_count How many async files wait for too long before execution.
# TYPE fslib_async_file_wait_timeout_count gauge
fslib_async_file_wait_timeout_count 0
# HELP fslib_fs_instance_count Count how many filesystem instances still alive
# TYPE fslib_fs_instance_count gauge
fslib_fs_instance_count 0
# HELP fslib_hdfs_jni_queue_len How many tasks are pending in the hdfs jni queue.
# TYPE fslib_hdfs_jni_queue_len gauge
fslib_hdfs_jni_queue_len 0
# HELP fslib_star_cache_memory_size How many memory bytes does star cache occupy. unit: B
# TYPE fslib_star_cache_memory_size gauge
fslib_star_cache_memory_size{fstype="cachefs"} 0
# HELP fslib_star_cache_meta_memory_size How many memory bytes does star cache's meta occupy. unit: B
# TYPE fslib_star_cache_meta_memory_size gauge
fslib_star_cache_meta_memory_size{fstype="cachefs"} 0
# HELP fslib_star_cache_disk_size How many disk bytes does star cache occupy. unit: B
# TYPE fslib_star_cache_disk_size gauge
fslib_star_cache_disk_size{fstype="cachefs"} 0
# HELP fslib_write_io_size Histogram of write io size, unit: bytes
# TYPE fslib_write_io_size histogram
fslib_write_io_size_count{fstype="cachefs"} 0
fslib_write_io_size_sum{fstype="cachefs"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="8192"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="16384"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="32768"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="65536"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="131072"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="262144"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="1048576"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="4194304"} 0
fslib_write_io_size_bucket{fstype="cachefs",le="+Inf"} 0
fslib_write_io_size_count{fstype="srposix"} 54490539526
fslib_write_io_size_sum{fstype="srposix"} 281895089487312
fslib_write_io_size_bucket{fstype="srposix",le="+Inf"} 54490539526
# HELP fslib_write_io_latency Histogram of write io latency, unit: microsecond
# TYPE fslib_write_io_latency histogram
fslib_write_io_latency_count{fstype="cachefs"} 0
fslib_write_io_latency_sum{fstype="cachefs"} 0
fslib_write_io_latency_bucket{fstype="cachefs",le="200"} 0
fslib_write_io_latency_bucket{fstype="cachefs",le="400"} 0
fslib_write_io_latency_bucket{fstype="cachefs",le="800"} 0
fslib_write_io_latency_bucket{fstype="cachefs",le="1600"} 0
fslib_write_io_latency_bucket{fstype="cachefs",le="3200"} 0
fslib_write_io_latency_bucket{fstype="cachefs",le="6400"} 0
fslib_write_io_latency_bucket{fstype="cachefs",le="12800"} 0
fslib_write_io_latency_bucket{fstype="cachefs",le="+Inf"} 0
fslib_write_io_latency_count{fstype="srposix"} 54490539526
fslib_write_io_latency_sum{fstype="srposix"} 340236322170.676
fslib_write_io_latency_bucket{fstype="srposix",le="+Inf"} 54490539526
# HELP fslib_read_io_size Histogram of read io size, unit: bytes
# TYPE fslib_read_io_size histogram
fslib_read_io_size_count{fstype="cachefs"} 0
fslib_read_io_size_sum{fstype="cachefs"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="8192"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="16384"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="32768"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="65536"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="131072"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="262144"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="1048576"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="4194304"} 0
fslib_read_io_size_bucket{fstype="cachefs",le="+Inf"} 0
fslib_read_io_size_count{fstype="srposix"} 142472457862
fslib_read_io_size_sum{fstype="srposix"} 997820557591628
fslib_read_io_size_bucket{fstype="srposix",le="+Inf"} 142472457862
# HELP fslib_read_io_latency Histogram of read io latency, unit: microsecond
# TYPE fslib_read_io_latency histogram
fslib_read_io_latency_count{fstype="cachefs"} 0
fslib_read_io_latency_sum{fstype="cachefs"} 0
fslib_read_io_latency_bucket{fstype="cachefs",le="200"} 0
fslib_read_io_latency_bucket{fstype="cachefs",le="400"} 0
fslib_read_io_latency_bucket{fstype="cachefs",le="800"} 0
fslib_read_io_latency_bucket{fstype="cachefs",le="1600"} 0
fslib_read_io_latency_bucket{fstype="cachefs",le="3200"} 0
fslib_read_io_latency_bucket{fstype="cachefs",le="6400"} 0
fslib_read_io_latency_bucket{fstype="cachefs",le="12800"} 0
fslib_read_io_latency_bucket{fstype="cachefs",le="+Inf"} 0
fslib_read_io_latency_count{fstype="srposix"} 142472457862
fslib_read_io_latency_sum{fstype="srposix"} 3633366540219.043
fslib_read_io_latency_bucket{fstype="srposix",le="+Inf"} 142472457862
# HELP fslib_close_io_latency Histogram of close io latency, unit: microsecond
# TYPE fslib_close_io_latency histogram
fslib_close_io_latency_count{fstype="cachefs"} 0
fslib_close_io_latency_sum{fstype="cachefs"} 0
fslib_close_io_latency_bucket{fstype="cachefs",le="200"} 0
fslib_close_io_latency_bucket{fstype="cachefs",le="400"} 0
fslib_close_io_latency_bucket{fstype="cachefs",le="800"} 0
fslib_close_io_latency_bucket{fstype="cachefs",le="1600"} 0
fslib_close_io_latency_bucket{fstype="cachefs",le="3200"} 0
fslib_close_io_latency_bucket{fstype="cachefs",le="6400"} 0
fslib_close_io_latency_bucket{fstype="cachefs",le="12800"} 0
fslib_close_io_latency_bucket{fstype="cachefs",le="+Inf"} 0r
Rocky
06/08/2026, 3:21 PMThe metrics you are looking for exist in your output, but they are named differently based on your StarRocks version and configuration. In shared-data clusters, many "Block Cache" metrics use the
default_cache_fslib_starrocks_be_block_cache_hit_bytesdefault_cache_hit_bytes0starrocks_be_page_cache_hit_countstarrocks_be_fslib_read_throughputfslib_read_io_size_sumiraterate Copy code
promql
irate(fslib_read_io_size_sum[1m])997 TB0fslib_open_cache_misses{fstype="cachefs"}fslib_read_io_latency_sumfslib_read_io_latency_countlake_delvec_cache_hit_minutelake_metadata_cache_hit_minutecachefsfslib_read_io_size_sum{fstype="srposix"}fstype="cachefs"0srposixstarrocks_be_page_cache_hit_count151 billionlake_delvec_cache_hit_minutelake_delvec_cache_missirate(fslib_read_io_size_sum{fstype="srposix"}[1m])Rocky
06/08/2026, 3:21 PMm
Minn Fong
06/08/2026, 3:25 PM@Rocky SO exactly which metrics should I be using to differentiate between cache hit vs S3 lookup
r
Rocky
06/08/2026, 3:25 PMTo differentiate between a Cache Hit (reading from local NVMe/RAM) and an S3 Lookup (reading from remote object storage) in a shared-data cluster, you need to look at the
fstypefslib_fslib_read_io_size_sum{fstype="s3"}ossks3hdfsfslib_read_io_latency_sum{fstype="s3"}fslib_read_io_latency_count{fstype="s3"}fslib_read_io_size_sum{fstype="srposix"}997 TBstorage_root_pathdefault_cache_hit_bytesstarrocks_be_page_cache_hit_countstarrocks_be_page_cache_hit_countfslib_read_io_size_sum{fstype="srposix"}fslib_read_io_size_sum{fstype="s3"}fslib_read_io_size_sum{fstype="cachefs"}cachefssrposixs3fstype="s3"fstype="s3"m
Minn Fong
06/08/2026, 3:50 PM@Rocky Do I need to change datacache_disk_size to a non-zero value equivalent to how much NVMe in total I will have per CN node? Default 0 means only RAM used to cache 3 reads?
r
Rocky
06/08/2026, 3:51 PMNo, you do not strictly need to change
datacache_disk_sizedatacache_disk_size500G2Tdatacache_disk_size0starlet_star_cache_disk_size_percentstarlet_star_cache_disk_size_percent = 90datacache_disk_size1800Gdatacache_mem_sizedatacache_disk_sizestarlet...percentstorage_root_pathdatacache_disk_sizedatacache_disk_size = 1800G Copy code
sql
SELECT * FROM information_schema.be_configs
WHERE NAME LIKE "%datacache_disk_size%"
OR NAME LIKE "%starlet_star_cache_disk_size_percent%";m
Minn Fong
06/08/2026, 3:53 PM@Rocky
Copy code
# StarRocks Backend Node Configuration for Stats
# Based on <https://docs.starrocks.io/docs/cover_pages/shared_nothing_deployment/>
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-exports=java.base/sun.nio.ch=ALL-UNNAMED"
# Network configurations
be_port = 9060
heartbeat_service_port = 9050
brpc_port = 8060
# Backend node service configurations
be_host = 0.0.0.0
priority_networks = 172.30.0.0/16
# Storage configurations
storage_root_path = /mnt/starrocks/data1/be/storage;/mnt/starrocks/data2/be/storage;/mnt/starrocks/data3/be/storage;/mnt/starrocks/data4/be/storage
# Log configurations
sys_log_level = INFO
# Memory configurations
mem_limit = 95%
# Data Lake cache (accelerates reads from Iceberg via External Catalog)
# datacache_enable = true
# datacache_disk_size = 1610612736000
# datacache_meta_path = /mnt/starrocks/be/datacache/meta
# Page cache configurations
storage_page_cache_limit = 20%
disable_storage_page_cache = false
# Compaction configurations
base_compaction_num_threads_per_disk = 2
cumulative_compaction_num_threads_per_disk = 2
# Streaming load configurations
streaming_load_max_batch_size_mb = 2048
streaming_load_max_mb = 102400
# Reduce flushes (512 MB) => Reduce disk I/O
write_buffer_size = 536870912
# Wide-table optimization (512 MB) => Reduce RPC call frequency to BE
max_tablet_write_chunk_bytes = 536870912
# Parallelize the PK index "Apply" phase for your 16-core nodes
update_compaction_num_threads_per_disk = 4
enable_prefetch = true
# Routine Load configurations (Kafka ingestion)
max_consumer_num_per_group = 5
# Security configurations
enable_token_check = true
# Transaction configurations
max_runnings_transactions_per_txn_map = 100
# Allow each tablet to run compaction more frequently (default 120)
update_compaction_per_tablet_min_interval_seconds=30
# BE checks tablet for compaction more frequently
update_compaction_check_interval_seconds=10
# PKI index compaction parallelism (default 1)
pindex_major_compaction_limit_per_disk=4
# Limits each compaction task to 100 rowsets
max_cumulative_compaction_num_singleton_deltas=100
# Delete trash directory after 1 hour
trash_file_expire_time_sec=3600
# More compaction concurrency
compact_threads=12r
Rocky
06/08/2026, 3:53 PMWhen moving from Shared-Nothing to Shared-Data (Cloud-Native), the role of your BE/CN nodes shifts: local disks become a Data Cache for S3, and compaction is coordinated by the FE.
Here are the most important configurations to add or tune for a Shared-Data cluster to maintain high ingestion throughput:
1. Data Cache (Critical for Performance)
In shared data,
storage_root_pathstarlet_use_star_cache = truestarlet_star_cache_disk_size_percent = 90datacache_disk_size = 01610612736000datacache_mem_size10G5%compact_threads412max_cumulative_compaction_num_singleton_deltas = 100lake_compaction_stream_buffer_size_bytes = 4194304lake_pk_compaction_max_input_rowsets = 500enable_pk_index_parallel_compaction = truelake_enable_ingest_slowdown = truelake_compaction_score_upper_bound = 2000lake_service_max_retry_count = 10lake_s3_upload_part_size = 33554432 Copy code
properties
# Network
starlet_port = 9070
# Storage (Now used as Cache)
storage_root_path = /mnt/starrocks/data1/be/storage;/mnt/starrocks/data2/be/storage...
# Data Cache
starlet_use_star_cache = true
starlet_star_cache_disk_size_percent = 90
# Shared-Data Compaction
compact_threads = 4
max_cumulative_compaction_num_singleton_deltas = 100
lake_compaction_stream_buffer_size_bytes = 4194304
lake_pk_compaction_max_input_rowsets = 500
enable_pk_index_parallel_compaction = true
# Object Storage Stability
lake_service_max_retry_count = 10lake_compaction_max_tasks-1Rocky
06/08/2026, 3:53 PM• administration/management/compaction.md
• deployment/shared_data/feature-support-shared-data.md
• knowledge/admin_and_operation/shared_data_compaction.md
• using_starrocks/caching/block_cache.md
References
• administration/management/compaction.md
• administration/management/BE_configuration.md
• deployment/shared_data/feature-support-shared-data.md
• knowledge/admin_and_operation/shared_data_compaction.md
• using_starrocks/caching/block_cache.md
References
• administration/management/compaction.md
• knowledge/admin_and_operation/shared_data_compaction.md
• administration/management/BE_configuration.md
• administration/management/FE_configuration.md
m
Minn Fong
06/08/2026, 6:31 PM@Rocky Why
sum(rate(fslib_open_cache_hits[$interval])) / (sum(rate(fslib_open_cache_hits[$interval])) + sum(rate(fslib_open_cache_misses[$interval])))sum(rate(fslib_read_io_size_sum{fstype="srposix"}[$interval])) / sum(rate(fslib_read_io_size_sum[$interval])) * 100r
Rocky
06/08/2026, 6:31 PMThese two metrics show different values because they measure different layers and units of the caching system.
1. Operation Count vs. Data Volume
* Query 1 (
fslib_open_cache_hitsfslib_read_io_size_sumsrposixfslib_open_cache_...fdfslib_read_io_size_sum{fstype="srposix"}srposixfslib_read_io_size_sumsrposixsum(rate(fslib_read_io_size_sum{fstype="srposix"}[$interval])) / sum(rate(fslib_read_io_size_sum[$interval]))m
Minn Fong
06/08/2026, 6:38 PM@Rocky Which metric should I use to monitor shared data cluster compaction, update latency, transaction publish latency etc?
r
Rocky
06/08/2026, 6:38 PMTo effectively monitor a shared-data (cloud-native) cluster, you need to track metrics from both the FE (Leader), which coordinates tasks, and the CN/BE, which executes them.
1. Compaction Monitoring
In shared-data, compaction merges data in S3. A high "Compaction Score" indicates that the background merging is falling behind the ingestion rate.
Metric | Level | Description
starrocks_fe_tablet_max_compaction_scorestarrocks_be_lake_compaction_max_scorestarrocks_be_compact_threadscompact_threadsstarrocks_be_lake_compaction_countstarrocks_be_update_rowset_commit_apply_duration_usstarrocks_be_update_apply_queue_countstarrocks_be_update_primary_index_numstarrocks_fe_txn_publish_latency_msstarrocks_fe_txn_publish_schedule_latency_mspublish_version_worker_countstarrocks_fe_txn_publish_execute_latency_msstarrocks_be_fslib_read_io_latencystarrocks_be_lake_get_tablet_metadata_latency_msSHOW PROC '/compactions'SHOW PROC '/transactions'SELECT _ FROM information_schema.be_compactionsm
Minn Fong
06/08/2026, 6:47 PM@Rocky Where do I download the dashboard json file
r
Rocky
06/08/2026, 6:48 PMYou can download the official StarRocks Grafana dashboard JSON files directly from the StarRocks resource site. Since you are running a shared-data cluster, it is recommended to use the specialized shared-data templates rather than the general "All Architecture" one.
1. Shared-Data Specific Dashboards (Recommended)
* *Shared-Data General Dashboard*: Focuses on shared-data specific features like transaction publishing, vacuuming (S3 cleanup), and metadata caching.
* *Starlet Dashboard*: Focuses on the "Starlet" storage engine layer, including S3/Object storage I/O, latency, and throughput.
2. General Dashboards
* *All Architecture Dashboard*: A comprehensive dashboard that works for both Shared-Nothing and Shared-Data clusters, covering CPU, Memory, and Query stats.
* *Kubernetes Overview Dashboard*: Best if you are running StarRocks via the Kubernetes Operator.
How to Import:
1. Open your Grafana Web UI.
2. Go to Dashboards > New > Import.
3. Click Upload dashboard JSON file and select the file you just downloaded.
4. Select your Prometheus data source (e.g.,
starrocks_monitorm
Minn Fong
06/08/2026, 8:00 PM@Rocky Is starrocks_be_tablet_update_max_compaction_score metric applicable to shared data cluster as well?
r
Rocky
06/08/2026, 8:00 PMNo, the metric
starrocks_be_tablet_update_max_compaction_scorestarrocks_fe_tablet_max_compaction_scoretablet_update_max_compaction_scoreTabletManager0lake_starrocks_be_lake_compaction_countstarrocks_be_lake_compaction_max_scorestarrocks_be_lake_pk_compaction_duration_usstarrocks_fe_tablet_max_compaction_score > 100SHOW PROC '/compactions'SELECT * FROM information_schema.partitions_meta ORDER BY MAX_CS DESC;m
Minn Fong
06/08/2026, 8:07 PM@Rocky Which metric to monitor NVMe space usage
r
Rocky
06/08/2026, 8:08 PMIn StarRocks, the metrics you should use depend on whether the NVMe drive is being used for Data Storage (Shared-Nothing) or as a Data Cache (Shared-Data/Cloud-Native).
1. If using NVMe for Data Cache (Shared-Data Cluster)
In shared-data clusters, NVMe drives are typically used to cache data from S3/Object Storage locally for performance. Monitor these metrics:
Metric | Level | Description
starrocks_be_datacache_disk_used_bytesstarrocks_be_datacache_disk_quota_bytesstarrocks_be_fslib_star_cache_data_disk_sizestarrocks_be_block_cache_disk_used_ratestarrocks_be_disks_total_capacitystarrocks_be_disks_avail_capacitystarrocks_be_disks_data_used_capacity Copy code
promql
(starrocks_be_disks_avail_capacity / starrocks_be_disks_total_capacity) * 100 < 20 Copy code
promql
(starrocks_be_datacache_disk_used_bytes / starrocks_be_datacache_disk_quota_bytes) * 100 > 95node_exporternode_filesystem_avail_bytes2 Views