https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
s
You make a good case for it. I would worry that such a change would break existing ingestions that rely on defaults in the same subtle way. I suggest that you add the request to Apache Druid github issues and the developer/user discussion can occur there as well.
v
@Clint Wylie what do you think about this? What does the JSON indexer do?
c
today the JSON column doesn't support multi value strings at all, it currently breaks arrays out into separate elements
this is changing very soon though with JSON indexer supporting true array columns
which will also be a top level replacement for using multi value strings
these will just hold arrays directly as they are
I want to move towards a world where people only use multi value dimensions when they truly never care about the array, and use arrays otherwise since they are so much more sane to reason about
v
that makes sense, especially if we couple it with data like 
["a", "b", "c"]
being defaulted to array (like in the data loader wizard) and people can explicitly ask for it to be a MV dimension
c
the nested column arrays will have indexes for both array_contains (and probably overlap) style operations, as well as equality when searching for specific arrays
all this said, I do agree the current defaults for multi value handling should probably be array, though we probably need to be careful thinking about if there are any implications for changing the default
which mode was used is also not persisted in the segment, so reingest doesn't necessarily use the same handling
which is another argument in favor of making plain array be the default
g
i would be good with making plain arrays the default. i agree it makes more sense as a default
for SQL-based ingestion it'd be also nice to have functions that sort and dedupe arrays like 
ARRAY_SORT(foo)
if you want 
foo
ingested as a sorted array
c
yeah, i agree about the sort and set operations so that it is possible to do most of the things mvds can do with arrays
id love to get the native 
map
,
fold
,
filter
etc array expressions wired up in sql somehow too so that then really could do anything with them
g
@Denis Ott are you interested in raising a PR? if so, link it here and we can continue the discussion there
d
@Gian Merlino I'll look into it, but I'm not familiar with the codebase and contribution guidelines yet, so it might take me a while. If someone steals the spotlight in the meantime i wouldnt mind
Open in Slack
PreviousNext
Powered by