https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, we're using Presto with Pinot and would l...
# general
a
Hi team, we're using Presto with Pinot and would like to support pushdown of functions like 
COALESCE
or multi-column 
CASE
statements on the Pinot side. This seems reasonable for predicates as currently it looks like push down logic is on aggregations / predicates. However, we're looking for some performance improvements here by having this as a 
SELECT
pushdown instead of having to return all data to Presto for processing as we can "aggregate" row-wise for various operators and take advantage of certain indexing i.e. bloom filters, etc. for 
COALESCE
, 
CONCAT
, etc. Are there concerns or pointers around this? @Xiang Fu
x
So far the support for pushing down different functions might need to be implemented separated.
If the semantics are the same across presto and pinot, you can try to have one general way to push down e.g. sum/count/min/max
thankyou 1
otherwise it may require extra override e.g. count(distinct xx)
for row level expression, so far there is pushing down for arithmetic, you can follow that to support more transform functions
You can check PinotPushdownUtils and PinotAggregationProjectConverter for more details
a
Awesome, thanks for the insight. Though one thing you mention is that arithmetic is pushed down, though I see in some query plans for example 
SELECT col1 + col2 FROM table LIMIT 10
results in a query plan that doesn't add them together but rather does the projection separate from the query itself?
Copy code
GeneratedPinotQuery{query=SELECT col1, col2 FROM table LIMIT 10...
            expr := (col1) + (col2) (1:80)
(@Nizar Hejazi by the way)
@Xiang Fu just to clarify on this front, what you mean by the pushdown of arithmetic is around aggregation, as aggregating across columns will push down properly, but not including an aggregation won't do that
So for the example above, if we wanted to do a sum of two columns for each row, the expectation is that we group by some unique identifier?
x
You can check TestPinotExpressionConverters.testProjectExpressionConverter
this is for converting the pushdownable expression with arithmetic
a
that specifies aggregation pushdown and I see when running explain plan that it pushes down only when aggregating, so one of the pre-requisite options is aggregation, but in the example above if I am just doing lets say 
SELECT col + 3 FROM table
this actually doesn't push down
@Xiang Fu want to follow up on this as pushdown is limited on this front
x
I think you are right, the transform itself won't be pushed down. The computation are left to presto . Pinot side is just data provider
2 Views
Open in Slack
PreviousNext
Powered by