only reason i’m calling this out is because i have been working on adding real array typed columns that behave consistent with more normal SQL array types, though they are still a bit experimental

https://druid.apache.org/docs/latest/querying/sql-data-types.html#multi-value-strings-behavior has some details about multi-value string behavior, though its missing the parts about the newer array columns since they are still experimental

array columns should be more fully featured in druid 27 release, which is coming soon

there are some differences between how multi-value columns and array columns are stored, and multi-value columns are limited to string type

internally multi-value string columns are stored a lot like regular string columns, both of which are dictionary encoded with 3 main parts, the column itself composed of the dictionary ids, the value dictionary which can lookup dictionary ids, and bitmap indexes corresponding to which rows of the id column contain which values

the dictionary id column part of multi-value columns stores arrays of ints instead of individual int values

so seeking to specific rows is a bit more complicated of an operation

in terms of code it is basically https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/data/CompressedVSizeColumnarIntsSupplier.java vs https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/data/V3CompressedVSizeColumnarMultiIntsSupplier.java

array columns have lineage from nested columns

nested columns have sequentially stacked value dictionaries for each primitive type of value, so a string dictionary for all the string values, a long dictionary, double dictionary, and then an array dictionary which stores int[] whose values are pointers into the other dictionaries

https://github.com/apache/druid/issues/12695 has the design on full nested columns, the new in druid 26 ‘auto’ schema has specializations for when there is no nesting, which is currently the only way to make array columns

so for array columns, instead of reading the row being an array of ints corresponding to dictionary id values, reading the row is a single int, which lookups into an int array, which each element is then looked up to get the element values

https://github.com/apache/druid/pull/13803 describes nested array columns, but ‘top level’ array columns are pretty similar

https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/nested/VariantColumn.java is the column implementation itself, currently it just shares with the mixed type column since they both need a lot of of the same stuff

disclaimer, filtering on array columns is basically broken in druid 26, but i hope it to be fixed in druid 27. it is not currently well wired up, but array columns have indexes both for whole array values and individual elements, while multi-value columns can only filter on individual elements, where if any value in the row matches the whole row matches (which can often create confusion of ‘how did that value match’ when coupled with the implicit UNNEST behavior that mvds have)

i guess the tl;dr is the biggest difference between mvds and arrays (besides the fact that arrays can distinguish

null

,

[]

, and

[null]

) is that rows for mvds are stored as

int[]

, which means the row is composed of multiple values, while rows of array columns store a single

int

meaning the entire array is a single value

nested columns are a bit more complicated so i glossed over it a bit, each path basically has a local dictionary which maps to the “global” dictionary space that is shared by all paths in the column

but the column of

'$.a'

would be like

[2, 1]

if it was two rows and the global dictionaries contained only those two values (and null, which is always 0)

the array dictionary stores int[] which contain the ids of the elements, so if the 3rd row was

{"a":[1]}

, the array dictionary would now have id

3

which stores`[2]` (assuming

2

is still the dictionary id of our

1

value (heh confusing :p)