https://pinot.apache.org/ logo
l

Laxman Ch

07/16/2021, 6:11 PM
Seeing some query failures and seeing the following errors in Pinot server intermittently
Copy code
2021/07/16 18:00:42.408 INFO [QueryScheduler] [pqr-1] Processed requestId=55381,table=domainEventView_REALTIME,segments(queried/processed/matched/consuming)=20/14/2/3,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=1,resSerMs=0,totalTimeMs=2,minConsumingFreshnessMs=1626458256041,broker=Broker_pinot-broker-1.pinot-broker.traceable.svc.cluster.local_8099,numDocsScanned=209,scanInFilter=940,scanPostFilter=426,sched=fcfs
2021/07/16 18:00:42.407 ERROR [ServerQueryExecutorV1Impl] [pqr-0] Exception processing requestId 55381
java.lang.RuntimeException: Caught exception while running CombinePlanNode.
java.lang.RuntimeException: Caught exception while running CombinePlanNode.
	at org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:157) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:157) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:33) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:33) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:294) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:215) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:157) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:33) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:294) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:215) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:157) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:157) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:294) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:141) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:215) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:157) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:141) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:141) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
	at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
	at org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:147) ~[pinot-all-hypertrace-0.7.1-5-shaded.jar:0.7.1-24ebc6d4cd25b1b21225f48c4e7438919246ffe3]
	... 15 more
Caused by: java.lang.NullPointerException
Any direction on how to debug this please?
k

Ken Krugler

07/16/2021, 7:31 PM
Looks like the last bit of the stack trace might be missing (which would show where the NPE is occurring)
l

Laxman Ch

07/16/2021, 7:35 PM
@Ken Krugler: This is the full log
j

Jackie

07/16/2021, 7:46 PM
It this the first time this is logged? The actual exception which causes the filure is the NPE, but there is no stack trace for that
b

Bruce Ritchie

07/18/2021, 8:19 PM
Try adding
-XX:-OmitStackTraceInFastThrow
to jvm args and retry.