https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, I ran into a weird issue when querying pi...
# troubleshooting
g
Hi team, I ran into a weird issue when querying pinot with trino. I am at pinot 0.10.0 (tried switch to 0.9.2, 0.8.0 and the error remains), trino v369, I have a table called 
usage_test
which showing good state in pinot and queryable through the pinot console, but when I run simple select query in Trino, like 
select * from pinot_1.default.usage_test limit 10
, query will fail with error like 
null value in entry: Server_pinot-server-2.pinot-server-headless.pinot.svc.cluster.local_8098=null
, but when I run aggregation queries like 
select count(*), hostname from pinot_1.default.usage_test group by hostname
query can succeed in Trino without issue. Does anyone have clue about this issue? Thanks in advance! 🙏
✅ 1
from the log it appears to me that seems like for plain select query trino wants to read in all the segments in this pinot table (
select *
) and then perform limit 10 on trino’s end? Is this expected?
Also I vaguely recalled I’ve seen some useful discussions in some channel about this issue before but I cannot find it anymore 😥, is it possible to extend the msg retention time of this slack workspace?
cc @Xiang Fu in case you have some insight 🙏
m
@Grace Lu The slack messages are sent to Apache Pinot mailing list, and should be searchable there.
g
@Mayank ah ty, let me look around the records there
👍 1
x
I think it’s due to the data table version issue
you can try to add 
pinot.server.instance.currentDataTableVersion=2
to pinot server
m
@Grace Lu ^^
g
@Xiang Fu yeah I found your comment about this in the prev threads and I already added this config, but it doesn’t seem to help in my case my current server config:
Copy code
root@pinot-server-1:/opt/pinot# cat /var/pinot/server/config/pinot-server.conf 
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
pinot.set.instance.id.to.hostname=true
pinot.server.instance.realtime.alloc.offheap=true

pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-east-1
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

pinot.server.instance.currentDataTableVersion=2
pinot.server.grpc.enable=true
pinot.server.grpc.port=8090
x
did you find any error log/exception in trino when you ran the query
g
yeah let me share more traces:
Copy code
2022-02-07T21:43:05.800Z	INFO	Query-20220207_214305_00031_uaicm-1476	io.trino.plugin.pinot.PinotSplitManager	Got routing table for usage_test: {usage_test_REALTIME={Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098=[usage_test__1__4__20211221T1221Z, usage_test__1__28__20211225T1222Z ..{Huge list of segments}..... usage_test__3__90__20220104T2032Z]}}
2022-02-07T21:43:35.834Z	ERROR	remote-task-callback-705	io.trino.execution.scheduler.PipelinedStageExecution	Pipelined stage execution for stage 20220207_214305_00031_uaicm.1 failed
java.lang.NullPointerException: null value in entry: Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098=null
	at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
	at com.google.common.collect.SingletonImmutableBiMap.<init>(SingletonImmutableBiMap.java:43)
	at com.google.common.collect.ImmutableBiMap.of(ImmutableBiMap.java:81)
	at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:126)
	at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:642)
	at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:620)
	at io.trino.plugin.pinot.PinotSegmentPageSource.queryPinot(PinotSegmentPageSource.java:221)
	at io.trino.plugin.pinot.PinotSegmentPageSource.fetchPinotData(PinotSegmentPageSource.java:182)
	at io.trino.plugin.pinot.PinotSegmentPageSource.getNextPage(PinotSegmentPageSource.java:150)
	at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:311)
	at io.trino.operator.Driver.processInternal(Driver.java:388)
	at io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)
	at io.trino.operator.Driver.tryWithLock(Driver.java:682)
	at io.trino.operator.Driver.processFor(Driver.java:285)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
	at io.trino.$gen.Trino_9c3bede____20220207_194140_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)


2022-02-07T21:43:35.835Z	ERROR	stage-scheduler	io.trino.execution.scheduler.SqlQueryScheduler	Failure in distributed stage for query 20220207_214305_00031_uaicm
java.lang.NullPointerException: null value in entry: Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098=null
	at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
	at com.google.common.collect.SingletonImmutableBiMap.<init>(SingletonImmutableBiMap.java:43)
	at com.google.common.collect.ImmutableBiMap.of(ImmutableBiMap.java:81)
	at com.google.common.collect.ImmutableMap.of(ImmutableMap.java:126)
	at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:642)
	at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:620)
	at io.trino.plugin.pinot.PinotSegmentPageSource.queryPinot(PinotSegmentPageSource.java:221)
	at io.trino.plugin.pinot.PinotSegmentPageSource.fetchPinotData(PinotSegmentPageSource.java:182)
	at io.trino.plugin.pinot.PinotSegmentPageSource.getNextPage(PinotSegmentPageSource.java:150)
	at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:311)
	at io.trino.operator.Driver.processInternal(Driver.java:388)
	at io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)
	at io.trino.operator.Driver.tryWithLock(Driver.java:682)
	at io.trino.operator.Driver.processFor(Driver.java:285)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
	at io.trino.$gen.Trino_9c3bede____20220207_194140_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
I tried to dig deeper into the problem and I think we might have some connectivity issue between trino worker and pinot-server on our end, didn’t think about this before because the aggregation queries work. I will look further on our end first and update here if it resolves, thanks guys for chiming in.
👍 1
x
ic, is your trino and pinot cluster deployed within same k8s?
✅ 1
4 Views
Open in Slack
PreviousNext
Powered by