Hey Pinot folks 👋 I'm trying to validate a PARTIAL upsert use case for our shipmentMerged table and ran into behavior I can't reconcile with the docs. Would love a quick sanity check on what's actually intended. Setup: mode: PARTIAL, nullHandlingEnabled: true, comparison column s_updated_at. Use case: dual-writer on the same table. One Flink pipeline emits full shipment records (all ~80 fields populated). A second pipeline emits partial records carrying only 2 NDR fields (is_ndr, no_of_attempts) — the other 78 columns are Avro-null. Goal: partial records update only the 2 NDR columns; full records update everything; intentional NULLs from source MySQL propagate to Pinot. What I observed on the test cluster:

Strategy   | Non-null incoming  | Null incoming (nullHandlingEnabled:        true)
-----------+--------------------+-----------------------------------------
IGNORE     | Keeps existing     | Keeps existing
           | (never updates     |
           |  after first       |
           |  insert)    
-----------+--------------------+-----------------------------------------       
OVERWRITE  | Takes new          | Framework preserves existing
           |                    | (null skipped before merger)

If this is correct, then: 1. There's no strategy combination that lets a single PARTIAL table support our dual-writer pattern AND propagate intentional NULLs from source. The two requirements collide on how "incoming null" is interpreted. 2. IGNORE is unusable for any column that ever needs to update — feels less like "ignore" and more like "freeze after first insert." 3. OVERWRITE + nullHandlingEnabled: true is effectively "OVERWRITE_NON_NULL" — useful, but it means partial-upsert can't intentionally clear a column to null. Questions: 1. Is the table above the documented intended behavior, or am I missing a config knob? 2. Is there a known pattern in Pinot for "update on non-null AND clear to null when source genuinely cleared"? (Other than nullHandlingEnabled: false, which loses Avro null semantics globally.) 3. Any plan for a null-aware variant strategy in upcoming releases? We've reverted to FULL upsert for our prod path. Just want to make sure I'm not under-using PARTIAL. Thanks 🙏 C.C: @Mayank @Xiang Fu @Jackie

there is a `FORCE_OVERWRITE`: newer strategy added by apache/pinot#15107, merged March 18, 2025, available from 1.4.0. This overwrites even when incoming value is null.

@Xiang Fu, but my partial upsert stream is coming from two separate pipelines, one via shipments where all the records are updated based on ID basis,(full upsert) and another stream, say NDR, coming with selective fields and in kafka these two are synced in two different formats, one with all the 84 fields and another with 10 fields(out of those 84 only) but other than these 10 others are null. When in upsert I dont want to ignore the nulls of shipments pipeline but I want to ignore the nulls of NDR(sink B). How can we achieve this?

this is not supported, you need to write your own merge logic