@Rocky What are the optimal settings for balancing ingestion (streaming writes via

stream_load

) and compaction configs on a ~50 node shared-nothing cluster? we're streaming in roughly 20K events per second across 18 tables. We see a lag accrue and either replication/cloning slow down our writes or compaction on nodes slowing down (reaching max tablet versions). are there any known production configs that need to be tuned?

References • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md • administration/management/BE_configuration.md

@Rocky the BE_configuration link doesn't work

Apologies for the broken link. The StarRocks documentation structure can sometimes shift between versions. You can find the current BE configuration page here: BE Configuration. For a ~50 node shared-nothing cluster handling 20K events per second across 18 tables, the "max tablet versions" error and write lag typically indicate that Cumulative Compaction cannot keep up with the frequency of your

stream_load

commits. Here are the recommended production configurations to balance high-concurrency ingestion and compaction in a shared-nothing environment: 1. Tune Compaction Concurrency (BE) In shared-nothing clusters, compaction is managed per-disk. If you have high-performance SSDs, the default single-threaded behavior is often the bottleneck. *

cumulative_compaction_num_threads_per_disk

: Increase this from

1

to

2

or

4

. This allows more parallel cumulative compaction tasks per disk. *

base_compaction_num_threads_per_disk

: Keep this at

1

or

2

. Base compaction is heavier; you don't want too many running at once. *

max_compaction_threads

: This is the global cap on a single BE. Set this to roughly 25% to 50% of your CPU cores (e.g., if you have 32 cores, set it to

8

or

16

). 2. Speed Up Compaction Tasks By default, StarRocks tries to merge up to 1000 rowsets in one task, which can take a long time and consume significant memory. *

max_cumulative_compaction_num_singleton_deltas

: Decrease this from

1000

to

100

. This forces the BE to run smaller, faster compaction tasks more frequently, preventing versions from piling up to the limit. 3. Ingestion (Stream Load) Optimization If you are hitting version limits, the most effective fix is often reducing the number of versions created in the first place: * Increase Batch Size: 20K eps is high if your batch sizes are small. Aim for at least 100MB to 500MB per

stream_load

or increase the interval between loads to

10 seconds

. *

streaming_load_max_batch_size_bytes

: Ensure this is high enough (default is 100MB, consider 500MB+ for high-throughput tables). 4. Backpressure and Version Limits If you still hit "too many tablet versions," you can increase the threshold before the system stops accepting writes: *

max_tablet_version_num

: Default is typically

1000

. You can increase this to

1500

or

2000

to provide a larger buffer, but this is a band-aid; it will eventually lead to higher query latency if compaction doesn't catch up. 5. Managing Replication/Cloning If replication/cloning is slowing down writes, it usually means nodes are under such high IO/CPU pressure that they are missing heartbeats or failing to report versions to the FE. * Check Disk IO: Use

iostat

to see if disks are saturated by compaction. If they are, lower the thread counts mentioned in step 1. * Check Memory: Compaction (especially on Primary Key tables) can be memory-intensive. Monitor

mem_limit

and OOMs. Summary of recommended

be.conf

changes:

properties
cumulative_compaction_num_threads_per_disk = 2
max_compaction_threads = 16
max_cumulative_compaction_num_singleton_deltas = 100

_(Adjust

max_compaction_threads

based on your actual core count per node.)_ References • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md • administration/management/BE_configuration.md