Hi all, I have a realtime table which completes lo...
# troubleshootingk
Kamal Chavda
11/12/2021, 6:41 PMHi all, I have a realtime table which completes loading all data from source ( using debezium > kafka). I compared the kafka connect logs and total records from snapshot match total records in Pinot table, however a few minutes later there are less records in Pinot. Nothing in the pinot-controller/server/broker logs. Anyone else experience this?
s
Subbu Subramaniam
11/12/2021, 6:51 PMDo you think the retention manager kicked in , removing old data? You may want to check the value of time column in your data. When you say "nothing" in the logs, I suppose you mean no errors/warnings?
k
Kamal Chavda
11/12/2021, 7:02 PMHi Subbu, yep, by "nothing" I meant errors/warning. Thanks for the suggestion, let me check that. If that's the case, increasing retentionTimeValue should work, correct?
Kamal Chavda
11/12/2021, 7:17 PMIt seems to be random. Compared Pinot table to source and I see missing primary key values in Pinot table. Where does the retention manager dump the removed data?