Hey, I have data around 50 Million in denormalized...
# general
p
Hey, I have data around 50 Million in denormalized form having year as a time partition, my ask is to aggregate the data irrespective of time, sample query : select a,b,c,d, sum(value) from table group by a,b,c,d limit 100 this query is taking a lot of time and sometimes recourse error as I understand it is aggregating to the granular levels, is there any way to improve the performance of this or by tuning any explain plan?
j
One possible tuning measure is to order the dimension columns so that a,b,c,d appear first. Druid will physically store the data in sorted order so lifting this would be faster.
j
Along the lines of what JRob is saying ... if you Range Partition on columns a,b,c,d and end up with only one range key in a given segment, you can gain some "perfect rollup" benefits which will reduce processing at the Broker level. https://druid.apache.org/docs/latest/ingestion/rollup#perfect-rollup-vs-best-effort-rollup