So, in total would I need both

RealtimeToOfflineSegmentsTask

and

MergeRollupTask

?

this config

"bucketTimePeriod": "1h",
        "bufferTimePeriod": "2h",
        "roundBucketTimePeriod": "1m",

will create segments containing 1 hour of data per segment. It won’t really be an

hourly rollup

because the

roundBucketTimePeriod

is

1m

. So from the data rows perspective, it will be a minute level rollup

So, in order to have an hourly rollup and daily rollup, the idea is to have :

"RealtimeToOfflineSegmentsTask": {
        "bucketTimePeriod": "1h",
        "bufferTimePeriod": "2h",
        "roundBucketTimePeriod": "1h",
        "mergeType": "rollup",
        "revenue.aggregationType": "sum",
        "maxNumRecordsPerSegment": "100000"
      }

yes this sounds right (assuming that when you say rollup, you want the time column to get rolled up to that granularity)

Thank you. 🙏 . Yes. I want the data to be rolled up to the time granularity windows.

@Neha Pawar If 10 day old data is ingested by a realtime table with a retention period of 5 days, will this data be handled by the realtime to offline task? Or will the data be cleaned up before the data can be moved to an offline table?

it will likely be cleaned up. The retention cleanup and realtimeToOffline are independent of each other. Retention cleanup happens every 6 hours, r2o happens as frequently as you’ve configured. so won’t be able to tell if it will be moved or deleted, but not something i would rely on.

Ok. In our situation we want to ingest 90 days of historical data to our realtime table and have this data be converted to offline segments. But we also want the realtime table to have a retention period of 1 day. To achieve this, would you advise to first create our realtime table without a retention period and once the data has been churned through and moved to offline segments we can add the 1 day retention value?

yes ^ . also depending on the frequency of the r2o job, you may want to have the retention be slightly more (if frequency is 1d, then atleast 3d retention. if frequency is 1h, then 1day retention is fine)

Alright that makes sense. Additional question, if data for a time period is queried and segments exist in both realtime and offline segments exist that cover this time period, what segments are used for the query? Do offline or realtime segments take precedence?