Hey 👋 , I am trying the upsert configuration to update a

TYPE

field for an event. I have followed the steps here https://docs.pinot.apache.org/basics/data-import/upsert. • set the primary key on the schema (unnested column from incoming json field) • have the upsertConfig set to

"upsertConfig": {
      "mode": "PARTIAL",
      "partialUpsertStrategies": {
        "type": "OVERWRITE"
      },
      "defaultPartialUpsertStrategy": "OVERWRITE"
}

• I also have the instanceSelectorType set

"routing": {
      "segmentPrunerTypes": [
        "partition"
      ],
      "instanceSelectorType": "strictReplicaGroup"
    }

• To test this, I produced an event with

status1

with the same primary key and republished the event with same id with

status2

. I expected to see only one record for this event with an updated (OVERWRITE)

status2

, but I see two events one with

status1

and other with

status2

. • What am I missing?

were the messages still processed on the same server or was the other partition on a different server?

Sorry don’t have much logs now, but I think they were on different servers.

ok, if that's right then it'd explain why. From my understanding, the upsert would only work if the messages are going to the same server. And actually you are right, it does need to be the same partition:

An important requirement for the Pinot upsert table is to partition the input stream by the primary key. For Kafka messages, this means the producer shall set the key in the send API. If the original stream is not partitioned, then a streaming processing job (e.g. Flink) is needed to shuffle and repartition the input stream into a partitioned one for Pinot's ingestion.

oh , I missed this somehow. thanks. but yeah makes sense to have them on the same partition.