https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
c
handling nested arrays of complex objects is still a bit of a work in progress, so i’m not sure if this can be done very easy or efficiently today
i’m planning on a couple of ways that this could be done, including wildcard support for path expressions, as well as supporting using 
UNNEST
, and longer term doing a more efficient internal representation of arrays of objects
unnest actually works at the native layer, but the SQL doesn’t recognize the json as something that returns an array of objects and so planning fails validation
but it wouldn’t be that efficient to extra values from those unnested objects since it would have to use the expression to just dig them out of the full object (compared to most 
json_value
expressions which have nearly the same performance as regular flat druid columns because we store nested columns for each field)
j
i see, thanks for the response. The good thing is i only use native instead of SQL, but seem like still don’t have a efficient way to achieve this.
How do you see if using a customized version complex type by writing my own extension?
This means i also need my own aggregators to unpack the customized type and aggregate the data.
c
yeah, its a decent amount of work, but might be appropriate if your needs are specialized
COMPLEX<json> was originally prototyped as an extension
👍 1
j
Would it be about same performance as using 
UNNEST
? Assuming 
UNNEST
is scanning the original data then unpack to get a unnest table, and then doing the actual aggregation, vs the customized version can unpack and aggregate while the scanning of original data .
c
yea, im sure you could do it more efficiently than it is done in druid that is currently released
once i get some more stuff done i imagine using a wildcard paths should be pretty efficient, since im imagining the wildcard paths will be able to pick out like 
weight
or other paths from the nested arrays of objects and spit out like 
ARRAY<LONG>
or whatever type the path is, and then that could be unnest to aggregate
you could also potentially just make a specialized aggregator that operates on the json to do the computation you need
well, you might need a specialized virtual column to extract the data, which is basically how json_value is able to work efficiently
arrays of objects aren’t really stored in the coolest way right now, basically there is a separate nested column per primitive field of the array element, and the element is part of the path (so in this example like 
$[0].weight
is a column and 
$[1].weight
is another column), which is kind of sad and inefficient in many ways, but otoh if you know what you want, it still should be relatively efficient to be able to get the values out of them for those specific fields
since like 
$[0].weight
is basically a regular long column
j
I see, thanks for the information.
once i get some more stuff done i imagine using a wildcard paths should be pretty efficient,
What is the time line for this?
c
not sure exactly, but feedback like this is totally useful to help me prioritize it
👍 1
i hope soon, since its really the last part that is kind of rough and arrays of objects happen often, so having a good story around doing stuff with them not involving specific elements is important
j
got it, thanks a lot for all the help and information.
m
Hey Clint, Following up on another question about this. You said that arrays of objects aren’t really stored in the best way right now, but wondering what if your data actually looked something like:
Copy code
{"driver":"Alice", "age":30, "vehicles_weights":{"Honda Civic":2877, "Toyata prius":3105}, "state": "CA"}
{"driver":"Bob", "age":27, "vehicles_weights":{"Honda CRV":3586, "Honda Civic":2877, "Toyata rav4":3640}, "state": "MA"}
{"driver":"Charlie", "age":40, "vehicles_weights":{"Honda CRV":3586, "Toyata prius":3005}, "state": "MI"}
{"driver":"Susan", "age":29, "vehicles_weights":{"Honda CRV":3586, "Toyata prius":3015}, "state": "MI"}
...}
Can you somehow sum all the weights across vehicles or do you still need to know all the keys and then use each key in 
json_value
expression in this case? Also Jianshu found

this video▾

that says with druid 24, there are significant impacts on storage and ingestion performance for nested data. Is that still true with druid 25?
Open in Slack
PreviousNext
Powered by