Hello @Rocky, everyone, We running a StarRocks cluster of 5 BE nodes, each 3.5 vCPU / 15 GiB mem_limit (All PK table with persistent index enabled). Our initial workload has run on this cluster comfortably for some time only 3 table with large in-frequent writes (1-2 a second). Now we have about 25 new table we aren't writing more data (in terms of rows count) but we are writing very often (only about 5 a second but we want about 100 a second to each table). So the question is how are we about to write to this cluster faster with out scaling it by 20X? During writes out CPU is at about +80% and memory is about 11 or 12 (out of 15). Using merge commit of 2000ms we were hitting a memory limit that would fail writes. A 500ms window was fine but only reduced CPU to about 60% for about 10 writes per second. I know doing external batching would help but would require a big engineering effort that we don't want to commit to right now so in the meantime Is there someway to juice up the merge commit? The memory allocated to loads is very small (~117Mib on average with peaks ~700Mib) but memory allocated to the update tracker is consistently at 6 Gib. Any ideas?

Your StarRocks cluster is currently facing a "memory squeeze" because 15 GiB of RAM is extremely tight for 25+ Primary Key (PK) tables with high-frequency writes. The 6 GiB allocated to the Update Tracker is likely the core issue, as it is competing with your data loads for the remaining memory. Here are the specific reasons and how you can "juice up" the performance without a 20x scale: 1. The "Smoking Gun":

l0_max_mem_usage

By default, each tablet (bucket replica) in a PK table with a persistent index can use up to 100 MiB for its in-memory Level-0 (L0) index. * The Math: If you have 25 tables, each with 10 buckets and 3 replicas, that is 750 tablets. * Even if only a fraction of those are "active" in memory, the default limits can easily consume your entire 15 GiB RAM. * The Fix: Reduce the per-tablet L0 memory limit in `be.conf`:

properties
  # Default is 104857600 (100 MiB). Try 10-20 MiB.
  l0_max_mem_usage = 20971520

Note: Reducing this will force the index to disk more frequently, which increases I/O but saves critical RAM. 2. Why Merge Commit is OOM-ing at 2000ms The

merge_commit

feature buffers data in memory before flushing. At 2000ms, your batch size is 4x larger than at 500ms. Since you only have ~3-4 GiB of "free" RAM (after the 6 GiB Update Tracker and other overheads), a large burst across 25 tables is likely hitting the

load_process_max_memory_limit_percent

(default 30% of RAM). Optimizations: * Keep the window at 500-1000ms but increase the compaction speed to keep the "Update Tracker" (specifically Delete Vectors) from growing. * Increase

tablet_max_versions

: High-frequency writes create many versions. If you hit the version limit, writes will fail or slow down.

properties
  # be.conf
  tablet_max_versions = 2000

3. Reclaiming "Update Tracker" Memory The Update Tracker memory stays high because it holds Delete Vectors and index metadata. These are only pruned during Compaction. If your CPU is at 80% and writes are frequent, compaction might be lagging. * Tune PK Compaction:

properties
  # be.conf
  update_compaction_num_threads_per_disk = 2
  update_compaction_per_tablet_min_interval_seconds = 60

* Bucketing Check: Ensure your tables aren't over-bucketed. For such small RAM and high table counts, you want the fewest buckets possible (e.g., 1-2 per BE) to minimize the metadata and index overhead. 4. Partial Updates (The Engineering Alternative) If your 100 writes/second are only updating a few columns of the row, use Partial Updates. * Standard PK writes send the whole row, which is memory-intensive. * Partial updates significantly reduce the data shuffled and stored in the write-ahead logs (WAL). Summary of Recommendations 1. Lower

l0_max_mem_usage

to 20MB in

be.conf

. This is the most likely way to shrink that 6 GiB Update Tracker. 2. Increase BE RAM to 32 GiB minimum if possible. 15 GiB is below the recommended minimum for production PK workloads with many tables. 3. Verify Tablet Count: Run

SHOW TABLET FROM <table_name>

and ensure you don't have hundreds of tablets per node. 4. Set

skip_pk_preload = true

in

be.conf

to prevent the BE from trying to load all PK indexes into memory on startup. Authority sources for verification: * StarRocks Primary Key Table Best Practices * Memory Management and Update Tracker Monitoring References • knowledge/trouble_shooting/fe_memory_problems.md • best_practices/primarykey_table.md • data_source/catalog/deltalake_catalog.md • release_notes/release-3.5.md