https://pinot.apache.org/ logo
#general
Title
# general
c

Carl

02/08/2022, 5:22 PM
Hi team, I have question about Pinot retention, is the retentionTimeValue changeable without recreating offline table? And what could be the reason segment with data older than retention period doesn't get deleted?
m

Mayank

02/08/2022, 5:25 PM
Yes, changing retention does not require recreating offline table. Retention job is a periodic thread, so it may take a few hours for it to be invoked (I think every 6 hours). Also, note that retention happens based on time column values in the segment, and not the time stamp when segment was created. So if your segment has even a single record that needs to be retained, the segment won’t be deleted.
c

Carl

02/08/2022, 5:28 PM
Thanks, I will check our segment logic.
👍 1
I spot checked a segment, it's max value of time column is 2020, and the time column is a INT 1DAYSEPOC datetimefieldspecs in table config. Not sure why it's still being retented. We have segmentsConfig.retentionTimeValue: 360 unit in days.
n

Neha Pawar

02/08/2022, 10:46 PM
any logs in the controller related to that segment fro the RetentionManager ?
this can happe if the startTime and endTime is not set correctly in the segment zk metadata. can you check those values in segment zk metadata via zk browser?
c

Carl

02/08/2022, 10:58 PM
Thank you for the information, we will check that.