https://pinot.apache.org/ logo
#general
Title
# general
c

Chundong Wang

11/10/2020, 9:16 PM
Just reported https://github.com/apache/incubator-pinot/issues/6253,
NullPointerException
is thrown when query with aggregation on top of groovy functions. Should we expect aggregation to work on top of groovy as it’s treated as transform function?
From the code, the NPE is thrown at,
Copy code
operatorLatch.countDown();
Which doesn’t make a lot sense to me as it should never be NULL…
c

Chinmay Soman

11/10/2020, 10:10 PM
might be better to continue in troubleshooting channel ? @Neha Pawar @Jackie FYI
n

Neha Pawar

11/10/2020, 11:27 PM
this works for me:
Copy code
select DestState, avg(groovy('{"returnType":"DOUBLE","isSingleValue":true}', 'arg0 > arg1 ? arg0.toDouble() : arg1.toDouble()', ActualElapsedTime, AirTime)) as theAvg from airlineStats where ActualElapsedTime>0 and AirTime>0 group by DestState limit 10
im on master though
nothing wrong with the query should’ve worked. let me investigate more
for 0.5.0 looks like the line throwing the NPE is
_indexedTable.finish(false);
any exceptions before this one?
c

Chundong Wang

11/10/2020, 11:50 PM
I don’t see other exceptions from the query response.
n

Neha Pawar

11/10/2020, 11:52 PM
how about in the server logs?
c

Chundong Wang

11/10/2020, 11:58 PM
let me rerun the query and see if I can find relevant log on brokers.
n

Neha Pawar

11/11/2020, 12:00 AM
please check server also. this exception is from the server code
c

Chundong Wang

11/11/2020, 12:00 AM
server you mean the nodes, right?
n

Neha Pawar

11/11/2020, 12:01 AM
i mean pinot-server logs. Check that along with pinot-broker
c

Chundong Wang

11/11/2020, 12:36 AM
I found below error on server,
Copy code
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
	at org.apache.pinot.core.operator.transform.function.GroovyTransformFunction.transformToDoubleValuesSV(GroovyTransformFunction.java:280) 
	at org.apache.pinot.core.operator.docvalsets.TransformBlockValSet.getDoubleValuesSV(TransformBlockValSet.java:85) 
	at org.apache.pinot.core.query.aggregation.function.AvgAggregationFunction.aggregateGroupBySV(AvgAggregationFunction.java:103) 
	at org.apache.pinot.core.query.aggregation.groupby.DefaultGroupByExecutor.aggregate(DefaultGroupByExecutor.java:143) 
	at org.apache.pinot.core.query.aggregation.groupby.DefaultGroupByExecutor.process(DefaultGroupByExecutor.java:130) 
	at org.apache.pinot.core.operator.query.AggregationGroupByOrderByOperator.getNextBlock(AggregationGroupByOrderByOperator.java:106) 
	at org.apache.pinot.core.operator.query.AggregationGroupByOrderByOperator.getNextBlock(AggregationGroupByOrderByOperator.java:38) 
	at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49) 
	at org.apache.pinot.core.operator.combine.GroupByOrderByCombineOperator$1.runJob(GroupByOrderByCombineOperator.java:137) 
	at org.apache.pinot.core.util.trace.TraceRunnable.run(TraceRunnable.java:40) 
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_265]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_265]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_265]
	at shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) 
	at shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) 
	at shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_265]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_265]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
Why is it casting things to double?
n

Neha Pawar

11/11/2020, 12:39 AM
2 reasons: 1. In your groovy query, i see return type double. So the final result returned by your script should be double. For instance, in the sample query I shared, my arg0 and arg1 were INT, so I used
toDouble()
on them 2. AVG, SUM functions expect double (i think)
it could be failing for you in one of these 2 palces
c

Chundong Wang

11/11/2020, 12:39 AM
AH
let me try converting everything to double first. 🙂
hmm… now i’m getting below error with
toDouble()
i think. Let me play around with it a bit. Thanks for the help!!
Copy code
java.lang.ClassCastException: java.math.BigDecimal cannot be cast to java.lang.Double
	at org.apache.pinot.core.operator.transform.function.GroovyTransformFunction.transformToDoubleValuesSV(GroovyTransformFunction.java:280)
	at org.apache.pinot.core.operator.docvalsets.TransformBlockValSet.getDoubleValuesSV(TransformBlockValSet.java:85)
	at org.apache.pinot.core.query.aggregation.function.AvgAggregationFunction.aggregateGroupBySV(AvgAggregationFunction.java:103)
I eventually get it to work. So to sum up, 1. Avg/Sum/PercentileTDigest## would all require
double
so need to convert the all possible return values to
double
; 2. I have a number literal,
0.0
, which I thought would make it double but actually is bigDecimal. So instead I used
0d
and second exception is gone.
n

Neha Pawar

11/11/2020, 1:33 AM
👌
c

Chundong Wang

11/11/2020, 1:33 AM
Much appreciated, @Neha Pawar!
🙂 1
🙏