Is there ever a use case to NOT use bitmap indexes...
# general
j
Is there ever a use case to NOT use bitmap indexes for a string dimension? I have a field with extremely high cardinality that is similar to a primary key. Is a bitmap index still useful in this scenario?
j
Index lookups use binary search, so you should filter on most select indexed columns first. This supports high cardinality columns being indexed. Furthermore, if you range partition on the high card column first, then the dictionary will get split across multiple segments, not duplicated across all segments, since the high card data is strictly sorted. This probably helps more in smaller segment sizes than index lookups. Last point ... there is recognition that sometimes your initial filters are selective enough that there is a point where you should stop doing filtered lookups, and instead render the rows and apply the final filters as a "row scan". There are some filter optimizations that were added earlier this year along those lines, and should automatically be used. (https://github.com/apache/druid/pull/16292)
✅ 1
c
you probably meant to link https://github.com/apache/druid/pull/15838, #16292 is a follow-up to adjust some behaviors of this one
the implication is yea that filter order can impact whether or not indexes are used, though we do intended on automatically ordering filters internally at some point based on likely complexity to compute
👍 1
its only available in druid 30+, before that change druid always uses indexes if they are available
j
Thanks!!