Hi Team, Observed one more case in RealTime Table. I was ingesting backdated data I.e from Jan 1,2022 yesterday. I had set retention to 15days. After sometime, segments were getting deleted, upon checking the logs. I got the following: 2022/01/24 070250.350 INFO [PeriodicTaskScheduler] [pool-10-thread-5] Starting RetentionManager with running frequency of 21600 seconds. 2022/01/24 070250.350 INFO [BasePeriodicTask] [pool-10-thread-5] [TaskRequestId: auto] Start running task: RetentionManager 2022/01/24 070250.352 INFO [ControllerPeriodicTask] [pool-10-thread-5] Processing 2 tables in task: RetentionManager 2022/01/24 070250.352 INFO [RetentionManager] [pool-10-thread-5] Start managing retention for table: reporting_aggregations_OFFLINE 2022/01/24 070250.358 INFO [RetentionManager] [pool-10-thread-5] Segment push type is not APPEND for table: reporting_aggregations_OFFLINE, skip managing retention 2022/01/24 070250.360 INFO [RetentionManager] [pool-10-thread-5] Segment lineage metadata clean-up is successfully processed for table: reporting_aggregations_OFFLINE 2022/01/24 070250.360 INFO [RetentionManager] [pool-10-thread-5] Start managing retention for table: reporting_aggregations_REALTIME 2022/01/24 070250.396 INFO [RetentionManager] [pool-10-thread-5] Deleting 4 segments from table: reporting_aggregations_REALTIME 2022/01/24 070250.396 INFO [PinotHelixResourceManager] [pool-10-thread-5] Trying to delete segments: [reporting_aggregations__0__0__20220124T1232Z, reporting_aggregations__0__1__20220124T1346Z, reporting_aggregations__0__2__20220124T1418Z, reporting_aggregations__0__3__20220124T1438Z] from table: reporting_aggregations_REALTIME 2022/01/24 070250.414 INFO [RetentionManager] [pool-10-thread-5] Segment lineage metadata clean-up is successfully processed for table: reporting_aggregations_REALTIME 2022/01/24 070250.414 INFO [RetentionManager] [pool-10-thread-5] Removing aged (more than 7 days) deleted segments for all tables 2022/01/24 070250.911 INFO [ControllerPeriodicTask] [pool-10-thread-5] Finish processing 2/2 tables in task: RetentionManager 2022/01/24 070250.911 INFO [BasePeriodicTask] [pool-10-thread-5] [TaskRequestId: auto] Finish running task: RetentionManager in 561ms 2022/01/24 070252.420 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 0 segments from table reporting_aggregations_REALTIME:[] 2022/01/24 070252.420 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 4 segments from table reporting_aggregations_REALTIME 2022/01/24 070256.457 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Moved segment reporting_aggregations__0__3__20220124T1438Z from hdfs://nameservice1/data/max/poc/hdfs/controller/reporting_aggregations/reporting_aggregations__0__3__20220124T1438Z to hdfs://nameservice1/data/max/poc/hdfs/controller/Deleted_Segments/reporting_aggregations/reporting_aggregations__0__3__20220124T1438Z 2022/01/24 070256.468 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Moved segment reporting_aggregations__0__0__20220124T1232Z from hdfs://nameservice1/data/max/poc/hdfs/controller/reporting_aggregations/reporting_aggregations__0__0__20220124T1232Z to hdfs://nameservice1/data/max/poc/hdfs/controller/Deleted_Segments/reporting_aggregations/reporting_aggregations__0__0__20220124T1232Z 2022/01/24 070256.468 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 2 segments from table reporting_aggregations_REALTIME:[reporting_aggregations__0__3__20220124T1438Z, reporting_aggregations__0__0__20220124T1232Z] 2022/01/24 070256.469 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 2 segments from table reporting_aggregations_REALTIME 2022/01/24 070258.353 INFO [PeriodicTaskScheduler] [pool-10-thread-6] Starting SegmentStatusChecker with running frequency of 300 seconds. 2022/01/24 070258.353 INFO [BasePeriodicTask] [pool-10-thread-6] [TaskRequestId: auto] Start running task: SegmentStatusChecker 2022/01/24 070258.355 INFO [ControllerPeriodicTask] [pool-10-thread-6] Processing 2 tables in task: SegmentStatusChecker 2022/01/24 070258.372 INFO [ControllerPeriodicTask] [pool-10-thread-6] Finish processing 2/2 tables in task: SegmentStatusChecker 2022/01/24 070258.372 INFO [BasePeriodicTask] [pool-10-thread-6] [TaskRequestId: auto] Finish running task: SegmentStatusChecker in 19ms 2022/01/24 070300.716 INFO [LeadControllerUtils] [ControllerLeadershipFetchingThread] Getting Helix leader: d9-max-insert-1.srv.net_9000, Helix version: 0.9.8, mtime: 1643035318065 2022/01/24 070304.488 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Moved segment reporting_aggregations__0__2__20220124T1418Z from hdfs://nameservice1/data/max/poc/hdfs/controller/reporting_aggregations/reporting_aggregations__0__2__20220124T1418Z to hdfs://nameservice1/data/max/poc/hdfs/controller/Deleted_Segments/reporting_aggregations/reporting_aggregations__0__2__20220124T1418Z 2022/01/24 070304.497 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Moved segment reporting_aggregations__0__1__20220124T1346Z from hdfs://nameservice1/data/max/poc/hdfs/controller/reporting_aggregations/reporting_aggregations__0__1__20220124T1346Z to hdfs://nameservice1/data/max/poc/hdfs/controller/Deleted_Segments/reporting_aggregations/reporting_aggregations__0__1__20220124T1346Z 2022/01/24 070304.497 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 2 segments from table reporting_aggregations_REALTIME:[reporting_aggregations__0__2__20220124T1418Z, reporting_aggregations__0__1__20220124T1346Z] 2022/01/24 070313.351 INFO [PeriodicTaskScheduler] [pool-10-thread-7] Starting RealtimeSegmentValidationManager with running frequency of 3600 seconds. 2022/01/24 070313.351 INFO [BasePeriodicTask] [pool-10-thread-7] [TaskRequestId: auto] Start running task: RealtimeSegmentValidationManager 2022/01/24 070313.352 INFO [ControllerPeriodicTask] [pool-10-thread-7] Processing 2 tables in task: RealtimeSegmentValidationManager 2022/01/24 070313.352 INFO [RealtimeSegmentValidationManager] [pool-10-thread-7] Run segment-level validation 2022/01/24 070313.425 INFO [ConsumerConfig] [pool-1

Deletion is based on the time column value in the segments. For example, if all records are older than 15 days then those segments will be deleted. I am guessing you configured to consumed from

smallest

offset, and the data in your stream was older than 15 days

Yes. I was ingesting from Jan1,2022. I was assuming the retention was based on segment creation time.

it will check minTime maxTime in segment zk metdata, which uses the timeColumnName from the data

so if a segments max time is less then retention , it will be dropped?

that’s right