@Rocky, does batching many writes into a single transaction benefit compaction?

Yes, batching many writes into a single transaction significantly benefits compaction and is a recommended best practice for StarRocks. Here is why batching is beneficial: 1. Fewer Rowsets and Versions In StarRocks, every successful load transaction creates a new Rowset (a version of data). • Without batching: If you perform 1,000 small writes, the system creates 1,000 small rowsets. The compaction process must then work much harder to merge these 1,000 tiny files into larger ones. • With batching: If you combine those 1,000 writes into a single transaction, only one (or a few) larger rowset is created. This drastically reduces the number of files the compaction engine needs to track and merge. 2. Reduced Compaction Score StarRocks uses a Compaction Score to measure the health of a partition. A high score (typically >100) indicates that there are too many unmerged versions. • High write frequency causes the score to skyrocket quickly. • When the score is too high, StarRocks may slow down (delay commits) or reject new writes entirely to allow compaction to catch up. Batching prevents these "too many versions" errors. 3. Lower CPU and I/O Overhead Compaction is a resource-intensive background process that consumes CPU and Disk I/O. • Merging many tiny files is less efficient than merging a few large ones. • By batching writes, you shift the "merging" work to the ingestion phase (memory-side), which is more efficient than having the system perform repeated background I/O to compact thousands of small files. 4. Improved Query Performance Queries must often merge or filter data across multiple rowsets (especially in Primary Key or Unique Key tables). Reducing the number of rowsets through batching directly improves query latency because the engine has fewer data versions to scan and reconcile at read time. Recommendation If you are using Stream Load or Routine Load, aim for: • Batch Size: At least 100MB to 1GB per load, or hundreds of thousands of rows. • Frequency: No more than one write every 5–10 seconds per table if possible. If you have extremely high-frequency data, use a buffer (like Kafka or an application-level cache) to batch data before sending it to StarRocks. References • knowledge/usage/load/shared_data_batch_ingestion.md • best_practices/primarykey_table.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md