https://pinot.apache.org/ logo
Join Slack
Powered by
Hi <@UDQU92KBK> <@UDT7GFEG6> - Can you guys help w...
# general
p
Hi @Mayank @Jackie - Can you guys help with this below issue we are getting for one of our Pinot table (hosted on 3 servers) while querying :
Copy code
[
  {
    "errorCode": 425,
    "message": "java.net.UnknownHostException: <some-server>: Name or service not known\n\tat java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)\n\tat java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)\n\tat java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)\n\tat java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)"
  },
  {
    "errorCode": 427,
    "message": "3 servers [server-1, server-2, server-3] not responded"
  }
]
Health of these machines are perfectly fine. Also, getting this exception on Brokers
Copy code
Caught exception while sending request 1 to server: <some-server>, marking query failed
java.net.UnknownHostException: <some-server>: Name or service not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
	at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929) ~[?:?]
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519) ~[?:?]
	at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) ~[?:?]
	at java.net.InetAddress.getAllByName0(InetAddress.java:1509) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1368) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1302) ~[?:?]
	at java.net.InetAddress.getByName(InetAddress.java:1252) ~[?:?]
	at org.apache.pinot.shaded.io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at org.apache.pinot.shaded.io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:206) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:46) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:180) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:166) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at org.apache.pinot.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-f9b39eeac02eaa7a0001dc4a809b277213390392]
	at java.lang.Thread.run(Thread.java:829) [?:?]
What could be the reason of such an exception? Is it making TLS connection from Broker to Server and is failing or some cache load issue? How to fix this? We tried restarting, Controller and Broker instances, tried rebalancing Brokers and Servers, but issue still persists.
k
hi can you please share the hostnames that are displayed in Pinot Console Ui for each component also is this deployed on k8s?
p
Hi @Kartik Khare - It will be difficult for me to share the hostnames as its deployed on Walmart cloud and its very centric. But, if you can tell me , what info you are looking at, I maybe able to answer. Also, we have managed to solve this issue by update tags API by replacing that server node throwing 425 unknown host exception with an entirely new Server node and ran a rebalance of table. What I understand is this is just a workaround, but can you help, why I can not run a force rebalance?
Open in Slack
PreviousNext
Powered by