Hi I added upsert configuration in pinot table. "primaryKeyColumns": [ "DEVICE" ] "upsertConfig": { "mode": "FULL", "dropOutOfOrderRecord": true, "hashFunction": "NONE", "defaultPartialUpsertStrategy": "OVERWRITE", "enableSnapshot": true, "metadataTTL": 0, "deletedKeysTTL": 0, "enablePreload": true }, "routing": { "instanceSelectorType": "strictReplicaGroup" } "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "topic_name", "stream.kafka.broker.list": "kafka_broker", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "1h", "realtime.segment.flush.threshold.segment.size": "100M" }, select count(*) as c, device from table_name group by device order by c desc limit 10 Im getting output's more than 1. help me out plz

Upsert requires that any given key only exists in a single partition. Is your topic partitioned on device?

No its not partitioned on device. May I know how to do it.

Partitioning the topic must be done prior to Pinot ingestion. This can be done using a stream processor like Kafka Streams or Flink.

Number of partitions is 3 for the topic im using now. If i understand properly we need to keep one partitions for those topic to which we upsert the pinot table.

The number of partitions isn't important. What's important is that the set of keys in each partition is unique.

Okay I understand this concept, I like to go with realtime example. In my use-case i have lakhs of machine's coming into a table. I like to use machine name as a primary key. How to configure the kafka topic for this use case. my message will look like { "device": abc, "custom_column_1":"value_2", "custom_column_2":"value_2", } Should I need to change like this abc:{ "custom_column_1":"value_2", "custom_column_2":"value_2", } Or what should I do. If there was any document related to this, Im happy to read

if you have control over how the messages are published to the topic, then one option would be to use device as the message key

Okay thanks, let me try this method.