https://pinot.apache.org/ logo
Join Slack
Powered by
Hey team, we are having a pinot cluster using `rel...
# troubleshooting
a
Hey team, we are having a pinot cluster using 
release-1.3.0
deployed in k8s, now we recently faced an issue after cluster rotation/server restart where we got duplicate rows for our primary keys even for same segment docs, and this was not solved even after few hours of server rotation. We had to reload segments to solve this. We even replicated the issue in our dev cluster, where we got different results in same queries which eventually resolved after 4-5 hrs. Sharing the table config, schema, queries, results in the thread. Can someone please help in this, thanks.
Upsert table config:
Copy code
"segmentsConfig": {
      "replication": "3",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "31",
      "replicasPerPartition": "3",
      "timeColumnName": "insert_timestamp",
      "segmentPushType": "APPEND",
      "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
      "minimizeDataMovement": false
    },
    "taskTypeConfigsMap": {
        "UpsertCompactionTask": {
          "schedule": "0 */5 * ? * *",
          "bufferTimePeriod": "1d",
          "invalidRecordsThresholdPercent": "30"
        }
      }
    },
    "routing": {
      "segmentPrunerTypes": [
        "partition",
        "time"
      ],
      "instanceSelectorType": "strictReplicaGroup"
    },
    "query": {
      "timeoutMs": 10000
    },
    "instanceAssignmentConfigMap": {
      "CONSUMING": {
        "tagPoolConfig": {
          "tag": "DefaultTenant_REALTIME",
          "poolBased": false,
          "numPools": 0
        },
        "replicaGroupPartitionConfig": {
          "replicaGroupBased": true,
          "numInstances": 5,
          "numReplicaGroups": 1,
          "numInstancesPerReplicaGroup": 5,
          "numPartitions": 1,
          "numInstancesPerPartition": 5,
          "minimizeDataMovement": false,
          "partitionColumn": "id"
        },
        "partitionSelector": "INSTANCE_REPLICA_GROUP_PARTITION_SELECTOR",
        "minimizeDataMovement": false
      }
    },
    "upsertConfig": {
      "mode": "FULL",
      "consistencyMode": "NONE",
      "comparisonColumns": [
        "update_timestamp"
      ],
      "metadataTTL": 86400000,
      "dropOutOfOrderRecord": false,
      "enableDeletedKeysCompactionConsistency": false,
      "enablePreload": true,
      "deletedKeysTTL": 0,
      "enableSnapshot": true,
      "newSegmentTrackingTimeMs": 10000,
      "hashFunction": "MURMUR3",
      "defaultPartialUpsertStrategy": "OVERWRITE",
      "upsertViewRefreshIntervalMs": 3000,
      "allowPartialUpsertConsumptionDuringCommit": false
    },
PK -> id
Copy code
"primaryKeyColumns": [
    "id"
  ]
Copy code
select count(*), id from fact_test_table
where
order_date = '2025-12-27'
group by 2
having count(*) > 1
order by 1 desc
limit 1000
sometimes returned 
2
rows, sometimes 
493
rows ------------------------------------------------------------------ ------------------------------------------------------------------ when checked for a particular 
id
whose data was getting replicated
Copy code
select current_status, $docId, $segmentName, update_timestamp, insert_timestamp from fact_test_table
where
id = 1602849389
limit 100
sometimes 2 rows
Copy code
current_status  $docId  $segmentName  update_timestamp  insert_timestamp
APPROVED  14199 fact_test_table__12__11184__20251226T1830Z  1766775016000 1766775016000
DELIVERED 21035 fact_test_table__12__11184__20251226T1830Z  1766775666000 1766775016000
sometimes 1 row
Copy code
current_status  $docId  $segmentName  update_timestamp  insert_timestamp
DELIVERED 21035 fact_test_table__12__11184__20251226T1830Z  1766775666000 1766775016000
m
Server restart cannot lead to this situation. Are you sure you didn't try some other operations? It seems that one replica’s upsert metadata is corrupted. Add host name to your query and see which server
a
So this is what happened in prod, we added a tag due to which all components rotated once After this we saw this issue, then we reloaded the segments to fix this Later we again restarted server pods and again saw the same issue I did the same with dev setup and was able to see the same issue after rotating everything
for this id in dev it is still happening 1605595765
Copy code
select current_status, $docId, $segmentName, update_timestamp, $hostName insert_timestamp from fact_test_table
where
id = 1605595765
limit 100
server 0 and 1 are returning this:
Copy code
-- current_status   $docId  $segmentName    update_timestamp    insert_timestamp
-- DELIVERED    37766   fact_test_table__7__11279__20251228T1518Z   1766859143000   pinot-preprod-server-0.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
and server 2 is returning this:
Copy code
-- current_status   $docId  $segmentName    update_timestamp    insert_timestamp
-- DELIVERED    66460   fact_test_table__7__11275__20251227T1113Z   1766859143000   pinot-preprod-server-2.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
-- DELIVERED    37766   fact_test_table__7__11279__20251228T1518Z   1766859143000   pinot-preprod-server-2.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
m
Is this partial upsert?
Seems metadata server-2 got corrupted.
I don't see how it can happen though. What release are you running ?
a
no FULL upsert
release-1.3.0
can share some log files for broker/server/controller for a sample segment that was causing issues if that helps
m
This does not lead to rolling restart:
we added a tag due to which all components rotated once
Try restarting the server-2, see if that fixes the metadata
a
This does not lead to rolling restart
not tag, template metadata annotation
Try restarting the server-2, see if that fixes the metadata
did a segment reload, now it returns 1 row for all the 3 servers for the earlier query
m
Yeah so metadata somehow got corrupted (I don't know how that could happen though)
a
can 
"consistencyMode": "NONE",
cause this?
m
No
ack 1
a
in server logs for a segment, I can see: “Skip removing segment because metadata manager is already stopped” “Found 1703 primary keys not replaced when replacing segment” different pks Server 2: 0 → 87,582 keys Server 0: 592 → 88,122 keys Server 1: 0 → 56,725 keys
m
@deepthi ch any idea what could lead to metadata corruption?
a
So assuming the earlier segment reload fixed the metadata, again did rollout restart of the servers just now, to reproduce the issue. Query1 (order_date = ‘2025-12-27’)
Copy code
select count(*), id from fact_test_table

where

order_date = '2025-12-27'

group by 2
having count(*) > 1
order by 1 desc
limit 1000
sometimes returns 0 rows, sometimes 1 row the one row returned sometimes was -> 
1605595765
Copy code
select current_status, $docId, $segmentName, update_timestamp, $hostName insert_timestamp from fact_test_table
where
id = 1605595765
limit 100
result1
Copy code
current_status  $docId  $segmentName    update_timestamp    insert_timestamp
DELIVERED   37766   fact_test_table__7__11279__20251228T1518Z   1766859143000   pinot-preprod-server-1.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
result2 (server 2 seems to have corrupted data)
Copy code
current_status  $docId  $segmentName    update_timestamp    insert_timestamp
DELIVERED   66460   fact_test_table__7__11275__20251227T1113Z   1766859143000   pinot-preprod-server-2.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
DELIVERED   37766   fact_test_table__7__11279__20251228T1518Z   1766859143000   pinot-preprod-server-2.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
------------------------------------------------------------------ ------------------------------------------------------------------ Query2 (order_date = ‘2025-12-26’)
Copy code
select count(*), id from fact_test_table

where

order_date = '2025-12-26'

group by 2
having count(*) > 1
order by 1 desc
limit 1000
sometimes 1 row, sometimes 2 rows the different row returned sometimes was -> 
1600163850
Copy code
select current_status, $docId, $segmentName, update_timestamp, $hostName insert_timestamp from fact_test_table
where
id = 1600163850
limit 100
result1
Copy code
current_status  $docId  $segmentName    update_timestamp    insert_timestamp
DELIVERED   4641    fact_test_table__11__11179__20251226T1730Z  1766696574000   pinot-preprod-server-1.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
result2 (server 0 seems to have corrupted data now)
Copy code
current_status  $docId  $segmentName    update_timestamp    insert_timestamp
DELIVERED   4641    fact_test_table__11__11179__20251226T1730Z  1766696574000   pinot-preprod-server-0.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
DELIVERED   33148   fact_test_table__11__11177__20251225T1512Z  1766696574000   pinot-preprod-server-0.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
So for different ids, different servers are having corrupted data + server rollout restart is causing the issues
m
From the log message I see this in the code:
Copy code
// For Upsert tables when some of the records get dropped when dropOutOfOrderRecord is enabled, we donot
        // store the original record location when keys are not replaced, this can potentially cause inconsistencies
        // leading to some rows not getting dropped when reconsumed. This can be caused when a consuming segment
        // that is consumed from a different server is replaced with the existing segment which consumed rows ahead
        // of the other server
a
In the table 
dropOutOfOrderRecord
is false and the affected segments are not consuming as they are few days old
m
a
3rd one, else
I think it is due to this log
Copy code
2025/12/29 17:38:50.500 WARN [fact_order_details_v6_test_dr_v4_drtest1_REALTIME-7-ConcurrentMapPartitionUpsertMetadataManager] [HelixTaskExecutor-message_handle_thread_10] Failed to find validDocIds snapshot to add segment: ffact_order_details_v6_test_dr_v4_drtest1__7__11275__20251227T1113Z out of TTL, treating all docs as valid
actual table name is fact_order_details_v6_test_dr_v4_drtest1
m
Hmm, do you have TTL (I haven't checked your table config)
a
this part of code https://github.com/apache/pinot/blob/31724a7690b60c5c7fbcfcee024835e380b89e5f/pino[…]ot/segment/local/upsert/BasePartitionUpsertMetadataManager.java newer segment being loaded first, causing this to be OutOfTTL and no snapshot file I think
m
Hmm, can you check the log for snapshot file - was it created? If so, why is it missing? If not, why was it not created?
When you restart the server, do you do a graceful shutdown? Or just hard kill?
a
hard kill
just rollout restart of statefulset
m
Well, pod delete should still trigger graceful shutdown. Unless you give it very less time it, and snapshot is not created within that time
Check the log around shutdown. Also check for snapshot file creation
ack 1
a
So when I search this within 24 hrs
Copy code
"Persisted validDocIds for segment" |= "fact_order_details_v6_test_dr_v4_drtest1__7__11275__20251227T1113Z"
I see 2 instances when snapshot is persisted: (server 0 and server 1, not server 2)
Copy code
2025-12-29 14:25:31	
2025/12/29 08:55:31.630 INFO [ImmutableSegmentImpl] [fact_order_details_v6_test_dr_v4_drtest1__7__11280__20251229T0855Z] Persisted validDocIds for segment: fact_order_details_v6_test_dr_v4_drtest1__7__11275__20251227T1113Z with: 84997 valid docs


2025-12-29 14:25:31	
2025/12/29 08:55:31.694 INFO [ImmutableSegmentImpl] [fact_order_details_v6_test_dr_v4_drtest1__7__11280__20251229T0855Z] Persisted validDocIds for segment: fact_order_details_v6_test_dr_v4_drtest1__7__11275__20251227T1113Z with: 84997 valid docs
in the server 2, I see this:
Copy code
2025-12-29 14:25:31	
2025/12/29 08:55:31.581 WARN [fact_order_details_v6_test_dr_v4_drtest1_REALTIME-7-ConcurrentMapPartitionUpsertMetadataManager] [fact_order_details_v6_test_dr_v4_drtest1__7__11280__20251229T0855Z] Could not get segmentLock to take snapshot for segment: fact_order_details_v6_test_dr_v4_drtest1__7__11279__20251228T1518Z, skipping

2025-12-29 14:25:31	
2025/12/29 08:55:31.608 WARN [fact_order_details_v6_test_dr_v4_drtest1_REALTIME-7-ConcurrentMapPartitionUpsertMetadataManager] [fact_order_details_v6_test_dr_v4_drtest1__7__11280__20251229T0855Z] Missed taking snapshot for 5 immutable segments
And no final snapshot due to this I think
Copy code
2025-12-29 23:04:17	
2025/12/29 17:34:17.962 INFO [fact_order_details_v6_test_dr_v4_drtest1_REALTIME-7-ConcurrentMapPartitionUpsertMetadataManager] [pool-35-thread-2] Skip removing segment: fact_order_details_v6_test_dr_v4_drtest1__7__11275__20251227T1113Z because metadata manager is already stopped

2025-12-29 23:04:47	
2025/12/29 17:34:47.182 INFO [fact_order_details_v6_test_dr_v4_drtest1_REALTIME-7-ConcurrentMapPartitionUpsertMetadataManager] [pool-39-thread-5] Skip removing segment: fact_order_details_v6_test_dr_v4_drtest1__7__11275__20251227T1113Z because metadata manager is already stopped

2025-12-29 23:05:20	
2025/12/29 17:35:20.930 INFO [fact_order_details_v6_test_dr_v4_drtest1_REALTIME-7-ConcurrentMapPartitionUpsertMetadataManager] [pool-41-thread-4] Skip removing segment: fact_order_details_v6_test_dr_v4_drtest1__7__11275__20251227T1113Z because metadata manager is already stopped
c
QQ, is this data corruption seen for the cases of same pk and same timestamp or occuring with diff timecol values as well? I see that the 
update_timestamp
seems to be same in both rows in different segments. By any chance, did the validDoc id snapshot was deleted from any of the sever. This is stored in the segment path. It seems like from your logs, one server didn't persist the valid docids but the other servers were able to do that..
Copy code
current_status  $docId  $segmentName    update_timestamp    insert_timestamp
DELIVERED   4641    fact_test_table__11__11179__20251226T1730Z  1766696574000   pinot-preprod-server-0.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
DELIVERED   33148   fact_test_table__11__11177__20251225T1512Z  1766696574000   pinot-preprod-server-0.pinot-preprod-server-headless.pinot-preprod-lb.svc.cluster.local
a
same pk and same timestamp or occuring with diff timecol values as well
occurring with diff as well as share here
I see that the 
update_timestamp
seems to be same in both rows in different segments
have seen this in diff as well where both APPROVED and DELIVERED states exists with different timestamps
By any chance, did the validDoc id snapshot was deleted from any of the sever
no we didn’t manually deleted anything
It seems like from your logs, one server didn’t persist the valid docids but the other servers were able to do that..
As per the logs shared here , server2 failed to persist validDocs due to this
Copy code
2025-12-29 14:25:31	
2025/12/29 08:55:31.581 WARN [fact_order_details_v6_test_dr_v4_drtest1_REALTIME-7-ConcurrentMapPartitionUpsertMetadataManager] [fact_order_details_v6_test_dr_v4_drtest1__7__11280__20251229T0855Z] Could not get segmentLock to take snapshot for segment: fact_order_details_v6_test_dr_v4_drtest1__7__11279__20251228T1518Z, skipping

2025-12-29 14:25:31	
2025/12/29 08:55:31.608 WARN [fact_order_details_v6_test_dr_v4_drtest1_REALTIME-7-ConcurrentMapPartitionUpsertMetadataManager] [fact_order_details_v6_test_dr_v4_drtest1__7__11280__20251229T0855Z] Missed taking snapshot for 5 immutable segments
Hey @Chaitanya Deepthi Chadalavada @Mayank, picking this thread back up (duplicate rows on one replica after a server restart, fixed by a reload). Sorry it took a while. I found the root cause and raised a fix. When the snapshot pass can't get a segment's lock (e.g. during a reload), it skips that segment but still persists the TTL watermark. After a restart, that segment counts as out of TTL and is loaded from its stale or missing snapshot, so its old rows come back on that replica only. The fix holds back the persisted watermark for segments that missed their snapshot, so they're loaded normally after a restart. • Issue: https://github.com/apache/pinot/issues/19831 • PR: https://github.com/apache/pinot/pull/19832 @Chaitanya Deepthi Chadalavada I started with the snapshot-on-shutdown idea but moved away from it. Consumers are still running when the metadata manager stops, so a snapshot at that point can capture uncommitted state, and it doesn't help on a hard kill (our case). The PR leaves stop() unchanged and covers both graceful and killed restarts. Separately, a small minion fix: SegmentGenerationAndPushTask ignores an explicit append.uuid.to.segment.name=false, so deterministic segment names can't be used with it. • Issue: https://github.com/apache/pinot/issues/19819 • PR: https://github.com/apache/pinot/pull/19820 Would appreciate a review on both when you get time. Thanks!
thanks 1
👍 1
c
Thanks Akshay! Good catch, in general the snapshots we persist today after commit don't have some gaurds during restarts. Your PR can fix the stale watermark case. Also I think this PR might fix some of the stale and missed snapshot cases and reduce the chances of this issue wouldn't completely fix it. It will be available in next OSS release github.com/apache/pinot/pull/19028
a
Thanks Deepthi — that helps. By basically introducing a new variable, was trying to force affected segments onto the full load path when snapshots were missed, which avoids duplicate PKs after restart. Also aligns with this comment:
Copy code
// Persist TTL watermark after taking snapshots if TTL is enabled, so that segments out of TTL can be loaded with
  // updated validDocIds bitmaps. If the TTL watermark is persisted first, segments out of TTL may get loaded with
  // stale bitmaps or even no bitmap snapshots to use.
Do lmk if any changes/modifications needed here
c
Approved the PR: Also fyi, we don't recommend using compaction and metadataTTL together which you have in your table. There is a chance of duplicates showing up for a given period of time which is not right. This happens because the server pk metadata is gone and if the older segments get compacted first with stale snapshot, there is a chance of duplicates staying in for a period of time. So disabled that combo until the fix is inplace.
thanks 1
a
Thanks a lot, actions not running though for the PR Checking the compaction and metadataTTL case, haven't seen this yet. Also is the fix present on master (any issue/pr if present)
c
We don't have a PR for this fix yet. github.com/apache/pinot/pull/19057- This was the PR from which we blocked. As an extension of your PR, if we allow the cleanup to only look at 
_persistedWatermark
, compaction and cleanup will be looking at same watermark from snapshots. When replacement happens we might still see the doc coming back. But when a segment commits, it should eventually be fine!
a
Got it, I'll check this once at my end and get back. Also, is the fix duplication pr good to merge?
c
Just one comment, rest looks good
ack 1
a
Added protected 
getPersistedWatermark()
in the latest commit On the deletedKeysTTL point, I want to check if I understand the case. Example with 
metadataTTL
7 days and 
deletedKeysTTL
1 day 1. Key K is inserted with ts 100 in segment S1. 2. K is deleted with ts 200 in segment S2. 3. A day later the sweep forgets K: it's removed from the map and the delete record's bit is cleared. 4. A new record for K arrives with ts 150. K isn't in the map anymore, so it's accepted and K is visible. 5. The server restarts. S1 and S2 are still within metadataTTL, so they're added normally, not skipped. 6. The delete record (ts 200) is read again and wins over the ts 150 record, so K now shows as deleted on this replica. Replicas that didn't restart still show K. Without preload, step 6 always happens, since the regular add reads every row. With preload, it happens if S2's snapshot from after step 3 isn't on disk yet. Is this the case you meant? If so, expiring against the persisted watermark would delay it but not close it. I think this will need a separate design to fix?
c
Yes, this is the case I am talking about. It is also helpful while reloading the segments to not revive the keys which are deleted and invalidated. Today during reload we rely on the snapshots as reliable source of truth, and we safely assumed since they get generated pretty often but there can be some edge cases like this. So I kinda feel like it would make sense to have the metadata cleanup to sync with the watermark we are persisting with the snapshots across the replicas.
a
only concern was higher memory in case snapshots misses are often, not soo sure about this also for the case mentioned, without preload will always result in deleted K so that case wont be solved using this
This PR is merged, thanks a lot for help and guidance. Have some other ways to handle the reload path and deleteKeysTTL, will raise it separately in a different issue. Also checking the compaction and metadataTTL case, might need some help in that.
thanks 1
2 Views
Open in Slack
PreviousNext
Powered by