This message was deleted.
# troubleshooting
s
This message was deleted.
u
This is a virtual column whose output type is STRING_ARRAY.
And we have filter on this saying vc != null
if i remove the filter this works, but with filter it is failing with this exception.
I saw the code and saw that there is no handling for ARRAY in makeProcessorInternal ? we do not expect Vcs to be Array ?
g
i guess you are using
useSchemaDiscovery
or nested/json columns? afaik that's the only way you can get
ARRAY<STRING>
columns
either way, if you had some multi-valued string columns before, there would be some query rewriting to do, described in these doc sections: • https://druid.apache.org/docs/latest/ingestion/schema-design.html#type-aware-schema-discoveryhttps://druid.apache.org/docs/latest/ingestion/schema-design.html#migrating-to-type-aware-schema-discovery
the biggest implication is this first one (the change from multi-value dimensions to arrays):
Update any queries that use multi-value dimensions (MVDs) to use UNNEST in conjunction with other functions so that no MVD behavior is being relied upon. Type-aware schema discovery generates ARRAY typed columns instead of MVDs, so queries that use any MVD features will fail.
if you aren't prepared to embark on that then please avoid
useSchemaDiscovery
for now. we're building a smoother migration path for an upcoming release, so you can wait for that
u
thanks. Will try to go through this and see what we can do about it.
g
my feeling is that multi-value strings -> arrays is a pretty major migration, so if you didn't mean to do it on purpose, it would be best to switch back
u
Trying to understand : how is multi value strings is different than the Arrays ?
I think problem is we have a custom function which was returning a MVD.
we are on 0.21.1 [Trying to move to 26], which recent version we can fallback without breaking mvd behaviour ?
g
where do you use your custom function?
you should be OK on 26 if you aren't using the new array columns or type-aware schema discovery features; they are new & optional
if you aren't using those, & are still seeing this error, please share some additional info about your custom function and we'll try to help
u
We do define the schema for each column during ingestion, i am hoping which means no schema discovery taking place.
I will check the details of custom function and will update. But as far as i know it take a single value and generate List<String> from it.
so the return type we changed from ``ExprType.STRING_ARRAY` to
ExpressionType._STRING_ARRAY_
assuming both are same