I started seeing this error on the when querying o...
# troubleshooting
j
I started seeing this error on the when querying on the multi stage query engine
Copy code
pinot-broker-1  | 2025/03/17 09:29:15.943 ERROR [MultiStageBrokerRequestHandler] [jersey-server-managed-async-executor-3] Caught exception executing request 1557543877000000008: select * from BankCustomer  limit 10, Error dispatching query: 1557543877000000008 to server: 172.18.0.134@{37197,43469}
pinot-broker-1  | org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:199)
pinot-broker-1  | org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:96)
pinot-broker-1  | org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:191)
pinot-broker-1  | org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:288)
pinot-broker-1  | UNAVAILABLE: io exception
pinot-broker-1  | io.grpc.Status.asRuntimeException(Status.java:537)
pinot-broker-1  | io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
pinot-broker-1  | io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
pinot-broker-1  | io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
pinot-broker-1  | finishConnect(..) failed: Connection refused: /172.18.0.134:37197
pinot-broker-1  | finishConnect(..) failed: Connection refused
pinot-broker-1  | io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:166)
pinot-broker-1  | io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
pinot-broker-1  | io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
pinot-broker-1  | io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
pinot-broker-1  | finishConnect(..) failed: Connection refused
pinot-broker-1  | io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:166)
pinot-broker-1  | io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
pinot-broker-1  | io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
pinot-broker-1  | io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
pinot-broker-1  |
I am running pinot server on 172.18.0.134 as a docker instance and I have not exposed these ports {37197,43469}. what are these ports used for? is it a static port or dynamic one? @Jayesh Asrani @Mayank @Xiang Fu
I am running on v1.1.0
Copy code
services:
  pinot-server:
    image: apachepinot/pinot:1.1.0
    ports:
      - "8098:8098" # helixPort
      - "8097:8097" # adminPort
      - "8090:8090" # grpcPort
      - "4004:4004" # JMX_port
      - "42997:42997" # queryMailboxPort I doubt this mapping works
      - "39765:39765" # queryServerPort I doubt this mapping works
    command: StartServer -configFileName /opt/pinot/conf/pinot-server.conf
    volumes:
      - ./pinot-server.conf:/opt/pinot/conf/pinot-server.conf
      - ./logs:/opt/pinot/logs
      - pinot_server_data:/tmp/pinot/data
      # todo: volume mount data
    environment:
      - JAVA_OPTS=-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=4004:/opt/pinot/etc/jmx_prometheus_javaagent/configs/server.yml -Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx11G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc:/opt/pinot/logs/gc-pinot-server.log
    extra_hosts:
      - "zookeeper1:172.18.0.137"
      - "zookeeper2:172.18.0.138"
      - "zookeeper3:172.18.0.139"
volumes:
  pinot_server_data:
    driver: local
docker compose of my servers
Copy code
# Pinot Role
pinot.service.role=SERVER

# Pinot Cluster name
pinot.cluster.name=prod-cluster

# Pinot Zookeeper Server
pinot.zk.server=zookeeper1:2181,zookeeper2:2182,zookeeper3:2183

# Use hostname as Pinot Instance ID other than IP
pinot.set.instance.id.to.hostname=true

# Pinot Server Netty Port for queris
pinot.server.netty.host=172.18.0.132
pinot.server.netty.port=8098

# Pinot Server Admin API port
pinot.server.adminapi.port=8097

# Pinot Server Data Directory
pinot.server.instance.dataDir=/tmp/pinot/data/server/index

# Pinot Server Temporary Segment Tar Directory
pinot.server.instance.segmentTarDir=/tmp/pinot/data/server/segmentTar

pinot.server.segment.fetcher.auth.token=Basic *************
pinot.server.segment.uploader.auth.token=Basic *************
pinot.server.instance.auth.token=Basic *************
pinot server config
fixed it Issue: Pinot V2 queries failed with a
gRPC UNAVAILABLE: io exception
error, while Pinot V1 worked fine. Fix: Changing the container network mode to
"host"
on each VM resolved the issue. Possible Cause: Pinot V2 uses gRPC with dynamic ports, which weren’t mapped in the previous setup. This caused connection failures between containers. Using
"host"
mode ensured all ports remained accessible.