Hello guys, This is my table { "REALTIME": { ...
# troubleshooting
k
Hello guys, This is my table { "REALTIME": { "tableName": "transaction_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "minimizeDataMovement": false, "timeColumnName": "updated_at", "schemaName": "transaction", "replication": "1" }, "tenants": { "broker": "cp_frm", "server": "cp_frm" }, "tableIndexConfig": { "rangeIndexVersion": 2, "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "loadMode": "MMAP", "streamConfigs": { "streamType": "pulsar", "stream.pulsar.topic.name": "topic name", "stream.pulsar.bootstrap.servers": pulsar server "stream.pulsar.consumer.prop.auto.offset.reset": "smallest", "stream.pulsar.consumer.type": "lowlevel", "stream.pulsar.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder", "stream.pulsar.consumer.factory.class.name": "schema" }, "enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false, "aggregateMetrics": false, "nullHandlingEnabled": true, "columnMajorSegmentBuilderEnabled": true, "optimizeDictionary": false, "optimizeDictionaryForMetrics": false, "optimizeDictionaryType": false, "noDictionarySizeRatioThreshold": 0.85 }, "metadata": {}, "routing": { "instanceSelectorType": "strictReplicaGroup" }, "query": { "timeoutMs": 3000 }, "instanceAssignmentConfigMap": { "COMPLETED": { "tagPoolConfig": { "tag": "cp_frm_REALTIME", "poolBased": false, "numPools": 0 }, "replicaGroupPartitionConfig": { "replicaGroupBased": true, "numInstances": 0, "numReplicaGroups": 1, "numInstancesPerReplicaGroup": 0, "numPartitions": 2, "numInstancesPerPartition": 1, "minimizeDataMovement": false, "partitionColumn": "id" }, "partitionSelector": "INSTANCE_REPLICA_GROUP_PARTITION_SELECTOR", "minimizeDataMovement": false }, "CONSUMING": { "tagPoolConfig": { "tag": "cp_frm_REALTIME", "poolBased": false, "numPools": 0 }, "replicaGroupPartitionConfig": { "replicaGroupBased": true, "numInstances": 0, "numReplicaGroups": 1, "numInstancesPerReplicaGroup": 0, "numPartitions": 2, "numInstancesPerPartition": 1, "minimizeDataMovement": false, "partitionColumn": "id" }, "partitionSelector": "INSTANCE_REPLICA_GROUP_PARTITION_SELECTOR", "minimizeDataMovement": false } }, "upsertConfig": { "hashFunction": "NONE", "defaultPartialUpsertStrategy": "OVERWRITE", "upsertViewRefreshIntervalMs": 3000, "partialUpsertStrategies": {}, "enableSnapshot": true, "metadataTTL": 0, "deletedKeysTTL": 0, "enablePreload": true, "consistencyMode": "SNAPSHOT", "newSegmentTrackingTimeMs": 10000, "dropOutOfOrderRecord": false, "enableDeletedKeysCompactionConsistency": false, "allowPartialUpsertConsumptionDuringCommit": false, "mode": "PARTIAL" }, "ingestionConfig": { "continueOnError": false, "rowTimeValueCheck": false, "transformConfigs": [ { "columnName": "updated_at", "transformFunction": "NOW()" } ], "segmentTimeValueCheck": true }, "isDimTable": false } When data gets inserted during consumption, the data is visible. Once the segment gets committed the data is visible only when I put SET skipUpsert=true or when I reloaded segements. Can someone help on why this is happening and the solution it. pinot version - 1.2.0 @Mayank
m
Can you paste the actual query and response?
k
@Mayank 2025-07-28 183708.093  2025-07-28T130707.874945267Z stdout F 2025/07/28 130707.874 WARN [transaction_REALTIME-0-ConcurrentMapPartitionUpsertMetadataManager] [transaction__0__246__20250728T1100Z] Found 20 primary keys not replaced when replacing segment: transaction__0__246__20250728T1100Z for partial-upsert table. This can potentially cause inconsistency between replicas This is the warning message in logs Once the data got inserted Pic 1: Data is visible in the query One the segment got comitted The results are Pic2,Pic3 Data is not visible with direct query
IMG_20250728_185118.png,IMG_20250728_185203.png,IMG_20250728_185145.png
m
@deepthi ch ^^
ack 1
c
Hi Can you use the latest Pinot version: 1.3.0 and let me know if you still see the issue? There are some fixes that got merged after 1.2.0. For example on how we are tracking the segments for consistent view when the consistency modes are set (SYNC, SNAPSHOT and NONE) in an upsert table: https://github.com/apache/pinot/pull/13677
k
@Chaitanya Deepthi Chadalavada May I know what happens to the existing data if we move to new version we are not using S3 or Hadoop as deep store
c
As long as you have deep store copy, you should be good. How’s your deep store configured? Since it’s a upsert table, it might take some time for the server to come up. As the replication factor is 1, you would have some downtime on this table.