https://pinot.apache.org/ logo
Join Slack
Powered by
I am executing a query and getting following error...
# general
m
I am executing a query and getting following error, 
select length(mydata) from mytable where length(mydata) > 500 LIMIT 10
. Other queries are working fine. Wondering whether this a query issue or cluster setup / index issue?
Copy code
ERROR SplitRunner-0-38 com.facebook.presto.execution.executor.TaskExecutor Error processing Split 20201121_222456_00039_rmrgi.1.0.0-4 PinotSplit{connectorId=pinot, splitType=SEGMENT, columnHandle=[PinotColumnHandle{columnName=mydata, dataType=varchar, type=REGULAR}], segmentPinotQuery=Optional[SELECT mydata FROM mytable_REALTIME LIMIT 2147483647], brokerPinotQuery=Optional.empty, segments=[mytable__0__7__20201121T2220Z], segmentHost=Optional[Server_test-pinot-server-1.test-pinot-server-headless_8098]} (start = 2824864.565796, wall = 3 ms, cpu = 0 ms, wait = 0 ms, calls = 1): PINOT_UNCLASSIFIED_ERROR: Error when hitting host Server_test-pinot-server-1.sb-pinot-server-headless_8098 with pinot query "SELECT mydata FROM mytable_REALTIME LIMIT 2147483647"
k
is this with Presto/Pinot connector?
m
yes that is correct @Kishore G
k
Does it work if you run it directly on Pinot?
prestodb or prestosql
m
good thinking, just tried and it works in Pinot
k
looks like the connector is not pushing the udf down to Pinot
Copy code
SELECT mydata FROM mytable_REALTIME LIMIT 2147483647
its trying to pull everything out of Pinot
m
that query worked once earlier , So do you think I have to check Pinot presto connector settings?
k
which presto are you using
prestodb or prestosql
m
prestodb the one from incubator-pinot github chart
The query always works without where clause like 
select length(mydata) from mytable limit 500
@Kishore G, After clearing the caches for presto the query started to work as normal. Thanks for pointing the issue .
k
it might be working but its not efficient
m
Thats correct there is a latency comparing with query directly hitting pinot and executing via presto
I was expecting the same performance though
k
its basically pulling all the data from Pinot into Presto and doing the computation in Presto, you want most of the computation to run in Pinot
yeah, its a matter of telling Presto to push this function down to Pinot
@Xiang Fu is there a way to configure this or it requires a code change in the connector?
x
is mydata a multi-value column?
I think presto has a function called array_length?
m
@Xiang Fu mydata is a STRING with maxsize 10000
mapped as VARCHAR in presto
x
i see, I think string length method is not pushed down through presto
it requires code change to do that
current approach is that presto will read all the strings back and compute the string length
it’s definitely inefficient
m
Ohh ok , most of the other methods are available right? so far I face issue with length only. Trying others as well
yes it is extremely slow
I now installed pinot driver in superset and trying to split the diff types of queries
x
for aggregations, we pushed down them, but those string functions we don’t pushdown yet
split the diff types?
you mean register both presto and pinot tables?
m
The ones require join from other datasources I can use presto driver and the queries that can be served just by pinot I can use pinot.
yes correct
x
got it
m
direct Pinot queries are performing well
x
yes, that’s why we need function pushdown 🙂
m
Hope that will get implemented at some point soon, Also came across SQL Passthrough https://prestosql.io/blog/2020/05/25/pinot-connector.html
x
yes, prestosql allows to directly pushdown whatever your wrote to pinot. That is good for users know how pinot query works and syntax
m
Thanks will try and see
Open in Slack
PreviousNext
Powered by