Jaideep C
03/17/2025, 9:35 AMpinot-broker-1 | 2025/03/17 09:29:15.943 ERROR [MultiStageBrokerRequestHandler] [jersey-server-managed-async-executor-3] Caught exception executing request 1557543877000000008: select * from BankCustomer limit 10, Error dispatching query: 1557543877000000008 to server: 172.18.0.134@{37197,43469}
pinot-broker-1 | org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:199)
pinot-broker-1 | org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:96)
pinot-broker-1 | org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:191)
pinot-broker-1 | org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:288)
pinot-broker-1 | UNAVAILABLE: io exception
pinot-broker-1 | io.grpc.Status.asRuntimeException(Status.java:537)
pinot-broker-1 | io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
pinot-broker-1 | io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
pinot-broker-1 | io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
pinot-broker-1 | finishConnect(..) failed: Connection refused: /172.18.0.134:37197
pinot-broker-1 | finishConnect(..) failed: Connection refused
pinot-broker-1 | io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:166)
pinot-broker-1 | io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
pinot-broker-1 | io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
pinot-broker-1 | io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
pinot-broker-1 | finishConnect(..) failed: Connection refused
pinot-broker-1 | io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:166)
pinot-broker-1 | io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
pinot-broker-1 | io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
pinot-broker-1 | io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
pinot-broker-1 |
I am running pinot server on 172.18.0.134 as a docker instance and I have not exposed these ports {37197,43469}.
what are these ports used for? is it a static port or dynamic one?
@Jayesh Asrani @Mayank @Xiang FuJaideep C
03/17/2025, 9:36 AMJaideep C
03/17/2025, 9:47 AMservices:
pinot-server:
image: apachepinot/pinot:1.1.0
ports:
- "8098:8098" # helixPort
- "8097:8097" # adminPort
- "8090:8090" # grpcPort
- "4004:4004" # JMX_port
- "42997:42997" # queryMailboxPort I doubt this mapping works
- "39765:39765" # queryServerPort I doubt this mapping works
command: StartServer -configFileName /opt/pinot/conf/pinot-server.conf
volumes:
- ./pinot-server.conf:/opt/pinot/conf/pinot-server.conf
- ./logs:/opt/pinot/logs
- pinot_server_data:/tmp/pinot/data
# todo: volume mount data
environment:
- JAVA_OPTS=-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=4004:/opt/pinot/etc/jmx_prometheus_javaagent/configs/server.yml -Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx11G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc:/opt/pinot/logs/gc-pinot-server.log
extra_hosts:
- "zookeeper1:172.18.0.137"
- "zookeeper2:172.18.0.138"
- "zookeeper3:172.18.0.139"
volumes:
pinot_server_data:
driver: local
docker compose of my serversJaideep C
03/17/2025, 9:48 AM# Pinot Role
pinot.service.role=SERVER
# Pinot Cluster name
pinot.cluster.name=prod-cluster
# Pinot Zookeeper Server
pinot.zk.server=zookeeper1:2181,zookeeper2:2182,zookeeper3:2183
# Use hostname as Pinot Instance ID other than IP
pinot.set.instance.id.to.hostname=true
# Pinot Server Netty Port for queris
pinot.server.netty.host=172.18.0.132
pinot.server.netty.port=8098
# Pinot Server Admin API port
pinot.server.adminapi.port=8097
# Pinot Server Data Directory
pinot.server.instance.dataDir=/tmp/pinot/data/server/index
# Pinot Server Temporary Segment Tar Directory
pinot.server.instance.segmentTarDir=/tmp/pinot/data/server/segmentTar
pinot.server.segment.fetcher.auth.token=Basic *************
pinot.server.segment.uploader.auth.token=Basic *************
pinot.server.instance.auth.token=Basic *************
pinot server configJaideep C
03/17/2025, 12:00 PMgRPC UNAVAILABLE: io exception error, while Pinot V1 worked fine.
Fix: Changing the container network mode to "host" on each VM resolved the issue.
Possible Cause: Pinot V2 uses gRPC with dynamic ports, which weren’t mapped in the previous setup. This caused connection failures between containers. Using "host" mode ensured all ports remained accessible.