Hi team, I have question about Pinot retention, is...
# generalc
Carl
02/08/2022, 5:22 PMHi team, I have question about Pinot retention, is the retentionTimeValue changeable without recreating offline table? And what could be the reason segment with data older than retention period doesn't get deleted?
m
Mayank
02/08/2022, 5:25 PM
Yes, changing retention does not require recreating offline table. Retention job is a periodic thread, so it may take a few hours for it to be invoked (I think every 6 hours). Also, note that retention happens based on time column values in the segment, and not the time stamp when segment was created. So if your segment has even a single record that needs to be retained, the segment won’t be deleted.
c
Carl
02/08/2022, 5:28 PMThanks, I will check our segment logic.
👍 1
Carl
02/08/2022, 7:24 PMI spot checked a segment, it's max value of time column is 2020, and the time column is a INT 1DAYSEPOC datetimefieldspecs in table config. Not sure why it's still being retented. We have segmentsConfig.retentionTimeValue: 360 unit in days.
n
Neha Pawar
02/08/2022, 10:46 PM
any logs in the controller related to that segment fro the RetentionManager ?
Neha Pawar
02/08/2022, 10:47 PM
this can happe if the startTime and endTime is not set correctly in the segment zk metadata. can you check those values in segment zk metadata via zk browser?
c
Carl
02/08/2022, 10:58 PMThank you for the information, we will check that.