https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
l

Lars-Kristian Svenøy

04/26/2022, 2:29 PM
Hello everyone 👋 For upsert tables its specified that you need to partition your input stream (kafka) by the primary key. How is this achieved when working with composite keys? I'm also curious how this works together with the column partition map, I thought it had to correspond to the partition in kafka?
For example..
This is the segment partitioning
Copy code
"tableIndexConfig": {
    "loadMode": "MMAP",
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "myPartitionKey": {
          "functionName": "Murmur",
          "numPartitions": 40
        }
      }
    },
Messages in the kafka topic are partitioned on "myPartitionKey" but the schema defines the composite primary key
Copy code
"primaryKeyColumns": ["myPartitionKey", "anotherKey"]
Is this correct?
m

Mayank

04/26/2022, 5:24 PM
@User @User for comments.
j

Jackie

04/26/2022, 5:37 PM
@User Your config should work because as long as
myPartitionKey
is partitioned, the composite keys are also partitioned
The prerequisite for using upsert is that all the records of the same primary key located in the same streaming partition
l

Lars-Kristian Svenøy

04/26/2022, 9:17 PM
Great thank you. That is what I expected, and means it will work the way I have it now 👍