I have a hybrid table. What tasks should I create ...
# general
s
I have a hybrid table. What tasks should I create in order to have both daily and hourly rollups?
p
I am curious about this use case as in how are / will the roll ups used? Do you keep raw data for some time, and make only rolled up data available after some time - lets say today and yesterday is available as raw events, next 5 days is available at an hourly level granularity, and next three weeks is available at daily level granularity? This is mostly for my own curiosity. Thank you in advance.
s
yeah...that granularity is promised via https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task . But honestly, I am confused too
p
I am guessing you are focused on how to enable it right now. I personally don't know how to. But my question is more around - how do you actually use the data once your table has events / metrics rolled up? I am confused about that part.
s
So, time boundary should automatically be understood by pinot. Say you have hourly rollups/daily rollups/weekly rollups enabled... AFAIK, all the rollups will be present for all the data. So if your data is like for 90 days, you will have all these rollups for all 90 days i.e even for your 90th day, there will be hourly rollups for that day.
Now when you query a period of data, pinot would look if there are any complete inclusive rollups. Like if you select a random 30 days across months, it will look at all valid weekly rollups for that period and for remaining days, it will look at daily rollups..
This is what I understand, but would be great if someone confirms
🙏 1
👍 1