@Rocky how can I achieve <4s insert-to-read latency using shared-data setups? (FE+cn). I am encountering rapid backlog of tablets to merge

To achieve <4s insert-to-read latency in StarRocks shared-data (FE + CN) architecture while resolving compaction backlogs, you should focus on optimizing the Version Publishing phase and the Compaction Scheduling efficiency. In shared-data mode, data is written to object storage, and the FE must coordinate the publishing of new versions across the cluster. If compaction lags, the number of un-merged rowsets (versions) grows, which increases read latency and slows down the publish phase. 1. Optimize Version Publishing (FE) Starting from StarRocks v3.2.3, a critical optimization exists for high-frequency ingestion. This feature allows the FE to batch multiple transaction commits, significantly reducing the overhead of publishing versions to object storage. * Enable Batch Publish:

sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");

2. Tune Compaction Performance (CN & FE) A "backlog of tablets to merge" indicates that the default compaction settings cannot keep up with your ingestion rate. * Increase Compaction Parallelism (CN): Increase the number of threads dedicated to compaction on your Compute Nodes.

sql
    -- Execute on CN (via information_schema for dynamic update)
    UPDATE information_schema.be_configs SET value = "16" WHERE name = "compact_threads";

* Speed Up Individual Compaction Tasks (CN): Reduce the number of data files merged in a single task so they finish faster and release resources sooner.

sql
    UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";

* Increase FE Compaction Concurrency: The FE schedules compaction tasks. If your cluster is large, ensure it can send enough tasks simultaneously.

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1"); -- -1 lets FE calculate based on CN count

3. Tune Ingestion Threads (CN) To handle the "insert" part of the latency, ensure the CN has enough worker threads to flush data to object storage quickly.

sql
UPDATE information_schema.be_configs SET value = "32" WHERE name = "flush_thread_num_per_store";
UPDATE information_schema.be_configs SET value = "32" WHERE name = "number_tablet_writer_threads";

4. Optimize Bucketing and Metadata In shared-data mode, every tablet requires a metadata file in object storage. High tablet counts (caused by too many buckets or partitions) increase metadata I/O latency. * Bucket Size: Aim for 1GB to 5GB per tablet. If your tablets are much smaller (e.g., <100MB), reduce the bucket count in your

DISTRIBUTED BY

clause. * Check Backlog Status: Monitor how far behind compaction is for specific tables:

sql
    SHOW PROC '/compactions';
    -- Or check specific tablet status
    SELECT * FROM information_schema.be_cloud_native_compactions WHERE STATUS != 'OK';

Summary of Targets * Compaction Score: Use

SHOW PARTITIONS FROM <table_name>

to check the compaction score. For <4s latency, you should aim to keep the score below 10. * Object Storage Latency: Ensure your object storage (S3/OSS/GCS) is in the same region as your CNs to minimize the physical I/O latency of the "Write" and "Publish" phases. References • faq/shared_data_faq.md • knowledge/usage/load/shared_data_batch_ingestion.md • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_compaction.md

How does it perform compaction? How similar or dissimilar to clickhouse merges @Rocky

StarRocks shared-data compaction and ClickHouse merges share a fundamental goal—consolidating small, immutable data parts into larger ones to optimize read performance—but they differ significantly in scheduling, transactionality, and I/O patterns due to their architectural differences. 1. Architectural Comparison * StarRocks (Shared-Data): Uses a FE-driven, decoupled approach. The Frontend (FE) acts as the brain, calculating "Compaction Scores" for every partition. When a score is high, the FE dispatches a compaction task to an available Compute Node (CN). The CN pulls data from object storage (e.g., S3), merges it, and writes the results back to object storage. * ClickHouse: Uses an autonomous, local approach. Each server (in a standard setup) manages its own background merges for the

MergeTree

family of engines. The merging happens locally on the disk where the data parts reside. 2. Transactionality & Visibility * StarRocks: Treates compaction like a standard data load. Every compaction task is assigned a

TxnID

. It goes through a "Write → Commit → Publish" cycle. The new compacted data is only visible to queries once the FE publishes the new version metadata. This ensures strict consistency across the shared-data cluster. * ClickHouse: Swaps "Parts" atomically on the local filesystem. Once a merge finishes and the new part is written, the old parts are marked for deletion, and the new part becomes visible for subsequent queries. 3. Key Operational Differences Feature | StarRocks (Shared-Data) | ClickHouse (MergeTree) Scheduling | Centralized (FE schedules to CNs). | Distributed/Local (Each node manages itself). I/O Path | Network-bound (S3/Object Storage ↔️ CN). | Disk-bound (Local SSD/EBS). Scaling | Elastic: You can add CNs just to handle a compaction backlog without moving data. | Rigid: Merges must happen on the node holding that specific shard's data. Backlog Impact | High "Compaction Score" triggers ingestion throttling (Slowdown/Rejection) at the FE level. | High part count eventually triggers

Too many parts

errors, blocking inserts. Compaction Types | Cumulative (merges small versions) and Base (merging everything into a foundation). | Background Merges (generally size-tiered/Log-Structured). 4. Why you see a "Backlog" in StarRocks In a shared-data setup, a backlog often occurs because: 1. Network Bandwidth: The CNs are constrained by the throughput to your object storage (S3/OSS) while downloading/uploading rowsets. 2. Publish Latency: If you have thousands of tablets, the metadata update (Publish) phase at the FE can become a bottleneck. 3. Default Concurrency: The default

compact_threads

or

lake_compaction_max_tasks

might be too conservative for high-frequency ingestion. Recommendations for your <4s Target To bridge the gap and match ClickHouse-like merge speeds in a shared-data environment: * Parallelism: Unlike ClickHouse where you might be limited by local disk IOPS, in StarRocks you can scale CNs. Ensure

lake_compaction_max_tasks

is set to

-1

(adaptive) and increase

compact_threads

on CNs. * Sized-Tiering: Ensure you are using StarRocks v3.2+, which improved the Size-Tiered Compaction strategy for Primary Key tables, making it much more efficient at handling high-frequency updates similar to how ClickHouse handles

ReplacingMergeTree

. * Batching: Use

lake_enable_batch_publish_version

(v3.2.3+) to reduce the FE metadata overhead, which is the most common cause of "insert-to-read" latency spikes. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • faq/shared_data_faq.md References

@Rocky how does compact_theads affect performance overall? Since many of the tablets may be smaller, it may be more efficient to merge many at once. Also take advantage of data caching already wanted data from the cached. Also how does having cn replicas play into all this

In a StarRocks shared-data (FE+CN) setup,

compact_threads

and the surrounding configuration determine how quickly you can clear a tablet backlog to maintain low insert-to-read latency. 1. The Role of

compact_threads

The

compact_threads

parameter (default 4) on the CN controls the number of tablet-level subtasks running concurrently on that specific node. * Overall Performance: Increasing this allows more tablets to be merged simultaneously. However, each compaction task consumes CPU and memory. If set too high, it can compete with query performance. * Recommendation: Set

compact_threads

to 25% of your CN's CPU cores. For example, on a 16-core CN, set it to 4 or 8 depending on your query load. * Small Tablet Efficiency: To clear a backlog of many small tablets quickly, it is often better to make each task finish faster. Reduce

max_cumulative_compaction_num_singleton_deltas

to 100 (from the default 1000/500). This forces the CN to merge smaller "batches" of files, releasing memory and thread slots back to the pool more frequently. 2. Leveraging Data Caching In shared-data mode, StarRocks uses Data Cache (Block Cache) to avoid remote S3/OSS reads. * Compaction is Cache-Aware: When a CN performs compaction, it reads data from remote storage. If that data is already in the local Data Cache (e.g., from a recent insert or query), the CN will read it directly from the local disk. This significantly reduces the "Read" phase of compaction. * Cache Warming: After compaction writes a new, merged version to object storage, that new data is also populated into the cache. This ensures that the next query (and the next round of compaction) can take advantage of the local disk speed. * Performance Tip: Ensure your CNs have high-performance local NVMe/SSD storage for the cache to minimize the I/O bottleneck during these merges. 3. How CN Replicas (Scaling) Help In the shared-data architecture, adding more CNs ("replicas" of your compute layer) is the most effective way to scale compaction throughput because data is not tied to a specific node. * Adaptive Concurrency: The FE parameter

lake_compaction_max_tasks

defaults to

-1

, which automatically calculates the cluster-wide limit as Number of CNs × 16. Adding a new CN node immediately increases the number of concurrent compaction tasks the cluster can handle. * Work Distribution: Compaction tasks are scheduled at the partition level by the FE and then broken into tablet subtasks. The FE dispatches these subtasks to any available CN. If one CN is busy with queries, the FE can send compaction tasks to other idle CNs. * No Data Redundancy: Unlike shared-nothing where data replication increases compaction overhead, adding CNs in shared-data mode only adds processing power. Every CN can technically work on any tablet because they all point to the same object storage. Summary Configuration for <4s Latency To clear the backlog, apply these dynamic updates:

sql
-- On FE: Allow the cluster to scale tasks based on CN count
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");

-- On CN: Increase thread concurrency and speed up small merges
UPDATE information_schema.be_configs SET value = "8" WHERE name = "compact_threads"; -- Adjust to 25% of cores
UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";

If the backlog persists after these changes, the bottleneck is likely FE Publish Version latency. Ensure you have enabled

lake_enable_batch_publish_version = true

on the FE to allow multiple transactions to commit their metadata at once. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md

@Rocky is compaction merging one (or a couple) files into the larger ones? That seems really expensive compared to merge tree based compaction

StarRocks' compaction is not a naive "one-at-a-time" merge. It uses a Size-Tiered Strategy that is architecturally very similar to ClickHouse’s

MergeTree

logic, but with different terminology and a decoupled execution model. 1. "Tiered" vs "Incremental" Merging If StarRocks merged every new small file into a giant base file immediately, write amplification would be massive. Instead, it follows an LSM-tree tiered approach: * Cumulative Compaction (Like ClickHouse Background Merges): StarRocks batches many small "Rowsets" (data parts) of similar sizes into a single larger one. It doesn't touch the "Base" data yet. This is exactly like ClickHouse merging several small parts into a medium-sized part. * Base Compaction: Only when the cumulative files grow large enough or meet specific criteria are they merged into the final "Base" Rowset. This is the heavy-duty consolidation. * Size-Tiered Strategy: For Primary Key tables (v3.1+), StarRocks uses a Size-Tiered Compaction Strategy by default. It organizes files into levels based on size (e.g., Level 0: <10MB, Level 1: <100MB). Compaction only happens within a level until it overflows to the next. This minimizes I/O and CPU, preventing the "merging one file into a giant one" scenario you feared. 2. Why it might feel more expensive than ClickHouse While the algorithm is similar, the overhead is different due to the Shared-Data architecture: * The Network Round-Trip: ClickHouse merges happen on local SSDs. StarRocks CNs must download the files from S3/OSS, merge them in memory (or local cache), and upload the result back. This is why Data Caching is vital—if the data is in the CN's local NVMe cache, the speed matches local disk performance. * The Publish Phase: In ClickHouse, a merge is a local file swap. In StarRocks Shared-Data, every compaction must be committed to the FE's metadata (the "Version Publish"). If you have thousands of tablets and frequent merges, the FE metadata updates can become the bottleneck, not the merge itself. 3. Handling the "Backlog" Efficiently If you see a rapid backlog, it’s usually because the "batches" are too small or the threads are saturated. * Batch More, Not Less: Ensure

max_cumulative_compaction_num_singleton_deltas

is high enough (e.g., 100–200). If this is too low, the system performs many tiny merges, which increases FE metadata overhead. * Horizontal Scaling: Since CNs are "replicas" of compute, adding more CNs linearly increases your compaction throughput. In ClickHouse, you are limited by the I/O of the node holding that shard. In StarRocks, any CN can merge any tablet. * PK Size-Tiered Tuning: Check if

enable_pk_size_tiered_compaction_strategy

is

true

(it is by default in recent versions). This is the key to achieving that <4s latency for high-update workloads. Summary Comparison Feature | StarRocks Shared-Data | ClickHouse MergeTree Strategy | Size-Tiered (Multi-level LSM) | Size-Tiered (Log-Structured) Write Amplification | Low (due to Tiering) | Low (due to Tiering) Merging Units | Multiple Rowsets (Batched) | Multiple Parts (Batched) Data Locality | Remote (S3) + Local Cache | Local Disk Scaling | Add CNs (Elastic) | Add Shards/Nodes (Manual Rebalance) Verdict: If you are seeing a backlog, it’s likely a concurrency or metadata publish bottleneck rather than an algorithmic inefficiency. Ensure you have enabled Batch Publish on the FE and have enough

compact_threads

on the CNs to take advantage of the tiered merging. References • administration/management/compaction.md • administration/management/BE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md

@jeremy.simpson If you want to achieve <4s insert-to-read latency using shared-data setups, there are two things you need to aware: 1. Using 4.1+ version. 2. Enable merge_commit