Hi Everyone, i've facing one issue regarding segments retention, as i've used RealtimeTable.json example from pinot 0.10.0 docs "retentionTimeUnit": "DAYS", "retentionTimeValue": "5", but after 5 days segments still there in controller data path also in server data path. did i miss something in my table json.

Retention is a periodic job, and happens every few hours in the background. Also a segment is deleted only when all records are older than retention. And as a good practice you should always have explicit time filter and not rely on retention to serve data for a particular period.

Hi, segment is deleted when all records older than retention. then how retention work on realtime tables, as new data came on table continuously.

Segments are periodically sealed and committed in the background in real-time as well. Sealed segments are deleted as and when they become old

Hi, thankyou for reply , can you share an example table json if its possible, where segments get deleted from local/hdfs after retention time.

Any sample config in the docs page will have retention time and unit

Hi, for simplify our discussion, i've attached my table json. where in table ive almost 30million records here i facing issue of retention, as i understands retention means while query over table, i am expecting results with in retention time cause before that data will be deleted. is thier any gap in my understanding.

The retention in your config is 2 days. So if your data that you pushed has time stamp values (for the time column you specified) older than 2 days, the data will be deleted the next time the retention task is kicked in

can we trigger retention task manually? As my data not getting deleted.

If it is not getting deleted that means there are records in segment that are not old

sure , i'll check and let you know!

Tar zip is on deepstore, not on server. Segments are periodically flushed to disk based on whatever setting you specified in table config (flush threshold for rows)

Here ive set "realtime.segment.flush.threshold.rows": "0", that is why its not getting deleted from disk?

There are a few settings, I think you probably set either time or size

I am following this doc https://docs.pinot.apache.org/configuration-reference/table, unable to find any setting, except realtime.segment.flush.threshold.time

Yeah so when segment is around that size it will be flushed. You probably also have a time threshold

it means let say i've set 50M segments (which may contains 1million records) and set realtime.segment.flush.threshold.time to 1d so once segment 50M reached new segment created, now for previous segment will be on disk till 1d then its get deleted. Please correct me if i am wrong!

In you setting above 1d is threshold to flush in memory to disk. That is different from retention. If you want to delete after x days, then set the retention value and time to be that, those are different configs

Thankyou @Mayank for clearing out my questions, As im newbee on this so sometimes ask some silly doubts 😄

Now worries