Hi, we are seeing below errors intermittently on broker . but the ip address it has is for RT MM . when we checked request logs on RT MM that query got succeeded . any inputs why this issue is happening

java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[?:?]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) ~[netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.6.Final.jar:?]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) ~[?:?]

i have seen issues like this happen when there are more connections in flight than the OS can handle. you could try addressing it by increasing your value of

net.core.somaxconn

(on linux). also try using druid 29 if you are on an older version, since it has this PR which increases the accept queue size to match the

somaxconn

value, which is also helpful: https://github.com/apache/druid/pull/15596

Thanks @Gian Merlino for the details . We are on 25 planning to upgrade to 29 . any suggestions on how we can do blue green deployment for Druid upgrades . can we spin up separate cluster with same metadata but we won't run any ingestion tasks on new cluster . the only issue i see is when we have Real time Kafka ingestion .

yeah, typically ingestion is a reason why a totally isolated blue/green deploy might not work well

ok ..thank you

@Sampath Kumar Chilukuri ^