Hi Team, we are exploring the rollup functionality in Pinot. Is there a functionality to have data rollup in different granularities like 1Dday, 1Hour, 10mins ? Our use case is when the user queries over a longer time period like 1 week, it will hit the segments with rollup of 1 day and if the user queries over a shorter time period like 1 hour, it will hit the segments with roll up of 10mins.

https://dev.startree.ai/docs/pinot/recipes/merge-segments-realtime Can configure roll up data at different resolutions based on the age. So a query over a month would use daily data for over a week old and then 10 min interval data for the current weeks data. If you want to keep the whole query to be daily data then you would need to use lastwithtime aggregate and group by the date, this can be used to aggregate the 10 min data into daily.

Mentioned in ^^^ OSS Pinot merge & rollup has a capability of doing multi level rollup (e.g. rollup hourly, daily monthly)

<https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task>

@Peter Pringle we want to do like if its querying last month over 1 week period, it should retrieve the roll up table of 1 day and if its querying last month but over 1 day period, it should roll up 10 mins. Keeping different granularity of the data. Smth like datadog.

Put those other time columns as metrics in the schema. Then you can aggregate max and they will not be included in the rollup grouping columns.

Put those other time columns as metrics in the schema. Then you can aggregate max and they will not be included in the rollup grouping columns.

yep haha we are thinking of this as well. Good point

For the query rollup, you can do this without using rollup task. Use lastwithtime aggregate fn, and group by a column with the key cols, rounded time. Having a rollup task to compact small segments at your most granular interval is still a good idea for performance but not needed for the query.

yep understood thanks alot Peter