@Rocky For partial update row mode, I have different events that come in at different times (can be merged in memory if share same parent key), updating ~20 columns. Is it better to merge all of them into a single partial update column schema or split them by co-occurence? So there is "bin-packing" of column updates in the sense of minimal null updates. Right now im doing a single column update schema so merge commit can combine concurrent partial updates from my distributed system. Not sure which way to do partial updates. The strategy im doing now is giving each event a 90 second merge window to collect all possible columns related to same parent key.

For your scenario in StarRocks, your current strategy of merging updates in memory (pre-aggregating over a 90-second window) is generally the best approach for performance, especially when using Row Mode partial updates. Here is a breakdown of why this is the case and how to handle the "bin-packing" vs. "single schema" trade-off: 1. Why "Merge in Memory" Wins Partial updates in Row Mode use a "Read-on-Write" mechanism. For every partial update row you send, StarRocks must: 1. Locate the existing row. 2. Read it into memory. 3. Merge the new columns with the old ones. 4. Write the new version of the row. This process is expensive (Write Amplification). By merging multiple events for the same parent key in your own distributed system before sending them to StarRocks, you reduce the number of these "Read-Merge-Write" cycles to exactly one per window per PK. If you split them into separate "bin-packed" schemas, you might end up updating the same row multiple times, which significantly increases the load on the storage engine. 2. Handling the "NULL Update" Problem The main risk with a "single broad schema" is accidentally overwriting existing data with

NULL

. In StarRocks, if a column is included in the

columns

header of a load, StarRocks will update it. If your data contains a

NULL

, it will overwrite the database value with

NULL

. To avoid this while keeping a single broad schema: * Use JSON Format: This is the most flexible. If you use

format: json

and

partial_update: true

, StarRocks will only update the fields actually present in the JSON object. If a field is omitted from the JSON for a specific row, the existing value in the database remains unchanged. * Dynamic CSV Columns: If you use CSV, you would need to group rows by the exact set of columns they are updating and send them in separate requests (or separate blocks) with a

columns

header that matches only those columns. This is essentially the "bin-packing" you mentioned. 3. Comparison: Single Schema vs. Bin-Packing Strategy | Best For | Pros | Cons Single Broad Schema (via JSON) | Sparse updates across many columns. | Highest efficiency. Minimizes write amplification by updating each row once. | JSON parsing is slightly more CPU intensive than CSV. Bin-Packing (Splitting by Co-occurrence) | Fixed-schema events (like CSV) that never overlap. | Good if events for the same PK never appear in different "bins." | If the same PK appears in multiple bins, you trigger multiple expensive "Applies" in Row Mode. 4. Recommendation for Your Architecture Since you are updating ~20 columns and already have a 90s merge window: 1. Stick with Row Mode: Row Mode is optimized for cases where you update many columns in a single row or have small real-time batches. 2. Continue Merging in Memory: This is your strongest optimization. Reducing the "row count" sent to StarRocks is key. 3. Use JSON with Omitted Fields: Instead of sending

NULL

for columns that didn't change in your 90s window, simply omit those keys from the JSON object. This allows you to use a single "logical" schema while ensuring StarRocks only touches the columns that actually changed. 4. Leverage Merge Commit: Since you are in a distributed system, keep

enable_merge_commit = true

. This will batch the transactions on the StarRocks side, which reduces metadata overhead (IOPS) and helps prevent "too many versions" errors. Summary: Do not split by co-occurrence (bin-pack) if it results in more rows per PK. Instead, merge as much as possible in memory and use a format (like JSON) that allows you to omit columns so they aren't overwritten with nulls. References • sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_set.md