Hi Team, Need your help to join the dots and correct my understanding or any generic suggestion is welcome. We have a realtime table XYZ with below config values (partially provided): "segmentsConfig": { "replication": "1", "replicasPerPartition": "1", "segmentPushType": "APPEND", "segmentPushFrequency": "HOURLY", "retentionTimeUnit": "DAYS", "retentionTimeValue": "2" }, streamConfigs": { "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "1G" } We have not modified any config to alter the default purge job which runs every 6hrs. Case 1: table created at 020000 June 10 UTC. first record ingested at 021500 June 10 UTC second record ingested at 012000 June 11 UTC third record ingested at 022000 June 11 UTC Q1: Is it correct that the first record is expected to be deleted from the database sometime between 021500 and 081500 on June 13 UTC ? My current understanding is that the segment created at 021500 June 10 UTC will be closed at 021500 June 11 UTC (due to realtime.segment.flush.threshold.time = 24h) and then 2 days of retention will be calculated (due to segmentsConfig.retentionTimeValue = 2 [segmentsConfig.retentionTimeUnit=DAYS]). Now as per schedule of Retention Manager maximum another 6hrs before the record is permanently deleted from the table. Since second record should fall in the same segment as the first record(1G won't cross) and purged at same time as first record. Case 2: We created the table and records as in case 1, but we altered the segment retention to 1 day at 043000 June 12 UTC. Q2: Will Retention Manager drop the segment having first and second record anytime between 043000 - 103000 on June 12 UTC ? Q3: What happens if we restart Pinot at 010000 on June 12 UTC, does Pinot run Retention Manager immediately or wait another 6 hrs ? Thanks for your inputs in advance!

Retention is not based on ingestion time, but on the value in the specified Time Column. A segment is eligible for deletion iff all records are older than the retention time based on the Time column.

thanks for the response! We want to predict when exactly does a record gets purged, even if that is within a window of time.

May I ask why?

at the very least, we should be able to define the behaviour of our system.

That's what I am saying depending on an async task to define the behavior of your system is not a good design. If you want to control when the retention kicks in, you can explicitly invoke the retention task with api (check swagger)