Hi All, I am trying to enable "UPSERT" mode in REALTIME table config in pinot 0.8.0 and the table is not able to read the records send to kafka topic. No results are displayed in PINOT UI at all, it shows 0 records. Below is the config I added for Upsert: "routing": { "instanceSelectorType": "strictReplicaGroup" }, "upsertConfig": { "mode": "FULL" }, I could not find anything significant in the controller logs as well. But when I remove the UPSERT config and tried, then my RealTime Table is able to read the records and getting displayed in Pinot UI. Any idea why is this happening?

Just to confirm, have you defined a primary key in your schema?

yes

It may help if you post both your table config and schema

we are suspecting problem on kafka , we are trying to create topic with proper partitioning.Below is the schema: { "schemaName": "wxcanalytics", "primaryKeyColumns": ["orgId","reportId"], "dimensionFieldSpecs": [ { "name": "reportId", "dataType": "STRING" }, { "name": "orgId", "dataType": "STRING" }, { "name": "firstName", "dataType": "STRING" }, { "name": "lastName", "dataType": "LONG" } ], "dateTimeFieldSpecs": [ { "name": "pdate", "dataType": "STRING", "format": "1DAYSSIMPLE_DATE_FORMAT:yyyy-MM-dd", "granularity": "1:DAYS" } ] } Table config: { "tableName": "wxcanalytics_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "timeType": "DAYS", "schemaName": "wxcanalytics", "retentionTimeUnit": "DAYS", "retentionTimeValue": "7", "timeColumnName": "pdate", "replicasPerPartition": "1" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.topic.name": "bc_data", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.broker.list": "xxxxxx:xxxxx", "realtime.segment.flush.threshold.rows": "1000000", "realtime.segment.flush.threshold.time": "1h", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" }, "enableDynamicStarTreeCreation": false, "aggregateMetrics": false, "nullHandlingEnabled": false, "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "loadMode": "MMAP", "enableDefaultStarTree": false }, "metadata": { "customConfigs": {} }, "routing": { "instanceSelectorType": "strictReplicaGroup" }, "upsertConfig": { "mode": "FULL" }, "isDimTable": false }

can u try moving the primaryKeyColumns after dateTimeFieldSpecs?

I tried adding the above config, but once the table is getting added when I check in the Pinot UI in the table config the "hashFunction": "NONE" is not getting added. I also tried through edit table , that didn't work too. I am using pinot 0.8.0

@Yupeng Fu ^^

Also check server log to see if sth unusual