Hi team, I’ve a question about RealtimeToOfflineSegmentsTask. If I configure this task like this: If I’m not misunderstanding the properties, it means every time this task is executed, 1 hour data older than 24h will be removed from realtime table to offline table. If it’s the case, when I keep bucketTimePeriod the same, and change the task to be executed every 2 hours, will there be longer and longer time data in the realtime table not be moved to offline table? If stream data constantly comes in this realtime table.

Bucket time is min-max time for segment. Buffer time is look-back window to search for records in bucket interval

So no matter how often the task is executed, only 24h data keeps in realtime table?

Couple of pages explains the params here: https://dev.startree.ai/docs/pinot/recipes/real-time-offline-job#real-time-table https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows

message has been deleted

ah you already found it

I think I’m still not very clear what does it exactly mean by “at a time” or “for each run”.

you can schedule it to run (i.e.

schedule

) but you can also run it manually if you want

If so, when bucketTimePeriod is set 1h, and task runs every 2 hours, more and more data will be kept in realtime table.

AFAIK, this task doesn't delete data from the real-time table

I think I see what you mean. Data older than 25 hours will be deleted by retention task. “retentionTimeUnit”: “HOURS”, “retentionTimeValue”: “25"

yeh. But it is a good question - should the scheduling frequency and bucket time be the same

i can confirm that yea it moves but it doesn’t delete from RT