so like i don’t actually recommend using master right now because im getting ready to open a PR sometime soon (like within the next 24 hrs soon because it needs to go into druid 26) that changes a ton about ^ because its backwards incompatible

but also does a totally rad thing to optimize the way the serialization of ^ works to write out only what is necessary for the specific type of column

For context, parquet data can be stored as list (i.e. list of double) but for other types like tsv, csv, etc I can seem to get druid ingestion to make the value into a array.

Context part 2. I want to feed an array into an aggregate for example building a double sketch from an array of double. This is useful for when your data (i.e. exisiting druid table or external table) is already rollup with the values stored as an array.

Use case is for building/changing type of sketch on a table that is already roll up

ah its nested arrays,

[[3,4], [5,6]]

currently ends up something like

[{"list":[3,4]},{"list":[5,6]}]

i think

there was another bug previously too https://github.com/apache/druid/pull/13294

expression transform array handling was improved by https://github.com/apache/druid/pull/13947

btw, here is the follow-up PR that reworks the first array PR https://github.com/apache/druid/pull/14014

its a bit of a beast