@Rocky I have a distributed system stream loading partial updates to my primary key table. Different events pertaining to same record may arrive at different nodes, hence I can't merge them in memory. Does it make a difference having 1 partial update record that contains all the merged events vs different, non-homogenous stream loads. One stream load column set per schema type? Like click, viewability, and mouse over events count as 3 different stream loads, so can't leverage merge commit? What if these events are small volume? Does it make a difference towards compaction and data freshness and CPU utilization of my BE nodes if I use Kafka to route them to deterministic partitions via the auction ID of the event? So I can merge them in memory and use the same partial update column schema for all events

Yes, it makes a significant difference. Using Kafka to route events by a deterministic key (like

auction_id

) and merging them in memory before loading is generally the highly recommended pattern for StarRocks Primary Key tables. Here is the breakdown of how these two approaches impact your system: 1. Merge Commit & Homogeneity * The Limitation: StarRocks' Merge Commit (introduced in v3.4) only supports merging homogeneous loading requests. This means the parameters, including the column list (

columns: ...

), must be identical. * The Impact: If you send three different stream loads with three different column sets (e.g., one for

click

, one for

viewability

), they cannot be merged into a single transaction by StarRocks. Each will generate its own data version, increasing the total version count and overhead. 2. Compaction and Data Freshness * Write Frequency vs. Batch Size: StarRocks is optimized for larger batch writes. Sending many small, non-homogeneous stream loads creates a "small file" problem. * Compaction Pressure: Each small load creates a new version (rowset). If you have three separate streams, you are effectively tripling the number of versions compared to a single merged stream. This leads to: * Higher Compaction Score: BEs will spend significantly more CPU cycles constantly merging these small files into larger ones. * Write Throttling: If the compaction score becomes too high, StarRocks will either slow down ingestion (shared-data) or return "too many versions" errors (shared-nothing). * Query Performance: A high number of uncompacted versions slows down read queries because the BE must merge more rowsets at read-time to determine the final state of a PK record. 3. CPU and Resource Utilization * Transaction Overhead: Every Stream Load has a fixed RPC and transaction management overhead on the FE and BE. Merging 3 events into 1 record reduces this overhead by 66%. * Column vs. Row Mode: * Row Mode (default for partial updates) is better for small, real-time updates but consumes more memory during compaction. * Column Mode is more CPU-efficient for large batches with few columns. By pre-merging in Kafka, you can create larger batches that might leverage Column Mode more effectively. 4. Deterministic Routing (Kafka) Using Kafka to route by

auction_id

is a classic "sharding" or "pre-aggregation" strategy that solves the "different events on different nodes" problem. * In-Memory Merging: This allows you to buffer events for a few seconds (e.g., 5–30s) and send a single, fully populated record. * Consistency: It ensures that even if you use partial updates, you are doing so in the most efficient "micro-batch" format possible. Summary Recommendation Metric | 3 Heterogeneous Loads | 1 Merged Load (via Kafka) Merge Commit | Not supported (different schemas) | Supported Data Versions | High (1 per event type per batch) | Low (1 per batch) Compaction CPU | High | Low Data Freshness | High (instant) | Slightly lower (due to Kafka buffering) Stability | Risk of "Too Many Versions" | Very High Conclusion: Even if your volumes are currently small, the "Merged Load" approach via Kafka is much more scalable. It protects your BE nodes from CPU spikes during compaction and ensures that as your volume grows, you won't hit the version limit bottleneck of transaction/version limits. _For more details, see the StarRocks Primary Key Table Best Practices._ References • administration/management/compaction.md • best_practices/primarykey_table.md