This message was deleted.
# general
s
This message was deleted.
j
Yes, this should be easily done using manual compaction/reindexing ... in the future a more comprehensive lifecycle management feature may help automate this in a more elegant manner.
👍 1
v
Yes, a datasource can have mixed granularity segments in it. There are two things to note here: (1) you generally want segments for a certain granularity to represent data that starts and ends on that granularity. So the
month
segmented data should only start on a month boundary, as in you do not want to, say, reindex 53 days of data into month granularity as those would not align with month starts and end. (2) I am sure you have a good reason to do this but this comment is mostly for other people reading this: this approach of having less granular segments is not some "cheat code" to better performance. You might get better performance or you might do all the work to set it up and not get any perf gains. This is because Druid always wants to divide your data into ~5M rows per segment (or some number). So if you have day segments and you have 900M rows of data in 30 days in
day
segments you will have 900M / 5M ~= 180 segments, if you reindex to month you will have 900M / 5M ~= 180 segments - as you can see if your data is large enough you will still get the same amount of segments even if you go to a less granular segment granularity. This will only give you benefits if your data is small (but then do your really need to optimize it?) or if you couple the reindexing with other operations that reduce the data size.
c
i am also rolling up values as the data gets older (as i go from hour => day => month granularities), so there should be some benefit here.