All the broker, server instances are facing the following issue randomly

java.net.UnknownHostException: krux-zookeeper

krux-pinot-server2  |  at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName0(InetAddress.java:1519) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName(InetAddress.java:1378) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  | 2022/09/09 14:50:43.169 ERROR [StaticHostProvider] [Start a Pinot [SERVER]-SendThread(krux-zookeeper:2181)] Unable to resolve address: krux-zookeeper:2181

krux-pinot-server2  | java.net.UnknownHostException: krux-zookeeper

krux-pinot-server2  |  at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName0(InetAddress.java:1519) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName(InetAddress.java:1378) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  | 2022/09/09 14:50:45.049 ERROR [StaticHostProvider] [Start a Pinot [SERVER]-SendThread(krux-zookeeper:2181)] Unable to resolve address: krux-zookeeper:2181

krux-pinot-server2  | java.net.UnknownHostException: krux-zookeeper

krux-pinot-server2  |  at java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName0(InetAddress.java:1519) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName(InetAddress.java:1378) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  |  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]

krux-pinot-server2  | 2022/09/09 14:50:46.955 ERROR [StaticHostProvider] [Start a Pinot [SERVER]-SendThread(krux-zookeeper:2181)] Unable to resolve address: krux-zookeeper:2181

krux-pinot-server2  | java.net.UnknownHostException: krux-zookeeper: Name or service not known

krux-pinot-server2  |  at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1529) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName0(InetAddress.java:1519) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName(InetAddress.java:1378) ~[?:?]

krux-pinot-server2  |  at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]

but after restarting it is up for sometime and again restarting. I did a ping from the individual instances and the ping is working fine.

looks like your DNS is broken. How are you running pinot?

I am running all the pinot and zookeeper in docker inside the same network

@John Peter S - this certainly looks like non-pinot issue. The randomness also hints at so.