https://pinot.apache.org/ logo
#pinot-dev
Title
# pinot-dev
a

Atri Sharma

07/06/2021, 7:11 PM
@User @User I am looking to support nulls in aggregates (a common use case for us). Is there a place where I can get prior thoughts and research, and potential starting ideas?
m

Mayank

07/06/2021, 7:13 PM
@User there has been some work done with null support in the past, perhaps we can start from where that discussion ended cc @User @User
j

Jackie

07/06/2021, 10:27 PM
@User Does putting a null filter work for your use case? E.g.
SELECT SUM(col) FROM table WHERE col IS NOT NULL
?
The main reason why we didn't directly support nulls in aggregates is because of the performance overhead of per-value null check, and forcing us to use
Object[]
instead of primitive array
a

Atri Sharma

07/07/2021, 5:15 PM
Could we use a Sentinel value to represent NULL, per data type?
j

Jackie

07/07/2021, 5:49 PM
That is doable, but can introduce some backward incompatibility. Also, the per-value if check is not avoided. A better solution could be applying the null value bitmap for each aggregate before scanning the values. Currently we can only apply null value bitmap on query level, not on aggregate level
k

Kishore G

07/07/2021, 5:55 PM
Jackie, you mean we can apply bitmap at filter phase not aggregation phase?
a

Atri Sharma

07/07/2021, 7:13 PM
I think we can apply it inside the aggregation operator itself?
Should be a simple matter of maintaining a per aggregation bitmap (maybe within the AggregationFunction?)
k

Kishore G

07/07/2021, 7:46 PM
we need to think about that a bit
its better to have the block return that
projection block should have nullvector
j

Jackie

07/07/2021, 7:52 PM
I mean we need to apply bitmap at aggregation phase in order to support multiple aggregations with different null bitmap
a

Atri Sharma

07/07/2021, 8:09 PM
@User You mean add a bitmap to ProjectionBlock? That would be sweet
It would come in handy if we ever decided to support joins, too
k

Kishore G

07/07/2021, 8:29 PM
yes, it should be a bitmap from the projectionblock
a

Atri Sharma

07/07/2021, 8:40 PM
OK, this makes sense. Let me investigate this