This message was deleted.
# generals
Slackbot
05/04/2023, 8:46 PMThis message was deleted.
v
Vadim
05/04/2023, 10:35 PMI am not sure what you are talking about, I routinely see super high cardinality string dimensions, with millions and millions of values.
Vadim
05/04/2023, 10:36 PMIs your only concern the size of the index in the segment?
Vadim
05/04/2023, 10:37 PMif that is truly your concern you can turn off bitmap indexes for specific dimensions (that will have a perf hit on filtering on those dims)
Vadim
05/04/2023, 10:37 PMSet create bitmap index to False
Vadim
05/04/2023, 10:37 PMnumeric dimensions do not have bitmap indexes
m
Max Reinsel
05/04/2023, 10:43 PMThanks for the reassurance here!
I'm glad to hear that both strings and numeric dimensions can scale to that degree. I know the bitmap indexes are compressed but I haven't played with how compressed they can be for longer bitmaps. In my case numeric is a more natural representation of this dimension, and I don't think we'll usually filter to just one value, so the filter perf hit won't be too bad.
Max Reinsel
05/04/2023, 10:45 PMIs there any interesting interaction with rollup being true / false? not knowing too much about the implementation, with rollup enabled you'd have to partition / shuffle the data by dimension to eventually get to one row for that dimension combination.
c
Clint Wylie
05/05/2023, 2:49 AMnested columns and soon, new in druid 26, numeric columns created with ‘auto’ schema do have indexes; they use a cardinality based threshold to decide when to use or not use the indexes in query processing
Clint Wylie
05/05/2023, 2:50 AMhttps://github.com/apache/druid/pull/13977 has some low level details
Clint Wylie
05/05/2023, 2:51 AMdisclaimer that this stuff is new and somewhat experimental
Clint Wylie
05/05/2023, 2:56 AMbut based on the measurements will likely apply that strategy to classic string columns at some point in the future
Clint Wylie
05/05/2023, 2:57 AMalso since those auto numeric columns have bitmap indexes, they are larger than classic numeric columns, in exchange for faster filtering
Clint Wylie
05/05/2023, 2:59 AMat some point in the future i’d like to add pure range based numeric indexes as a middle ground for very high cardinality numeric columns instead of a bitmap per value, but haven’t gotten there yet