This message was deleted.

What version are you on? We have made a lot of improvements to coordinator in last year so if you are at a version prior to 27, I would recommend doing an upgrade first.

We have upgraded to 27 couple of days back and we see similar issues but I see little improvement

If you are still seeing problems, please open a github issue with the details about the issues you are facing.

Should we consider increase in peons from 1 to 2 in each middle manager and have less than 200 middle managers ? or its same in both the cases ?

Generally you only have one MM per data server ... how many data servers do you have?

It’s an kubernetes deployment

I guess you have combined coordinator+overlord. Maybe splitting them will help. Also consider extending coordination period

I was wondering the same thing. I'm using druid on kubernetes, and I'm using one or two middlemanagers with a lot of resources allocated to them. Is there any reason to use 200 middlemanagers?

currently the coordinator.poll period is configured as 120s @윤혁준 do i have to increase more ? we wanted to scale mm horizontally and failing of one peon shouldn't affect other peons. So do you recommend to have multiple peons in the same middlemanager would improve the inter process communication ?

set

druid.coordinator.period

high enought so runs can end before the next starts (+some margin)

I know that peons floating inside a middleman don't affect each other.

@Abhishek Agarwal Here is the common error we have been observing in the CoOrdinator process.

2023-10-21T19:29:59,965 INFO [ServiceClientFactory-3] org.apache.druid.rpc.ServiceClientImpl - Service [index_kafka_kafkaInline_a6f18e5c64dc530_mnbpnkjn] request [GET <http://192.168.234.144:8100/druid/worker/v1/chat/index_kafka_data5g_a6f18e5c64dc530_mnbpnkjn/status>] encountered exception on attempt #2; retrying in 4,000 ms (org.jboss.netty.channel.ChannelException: Faulty channel in resource pool)
2023-10-21T19:29:59,965 WARN [HttpClient-Netty-Boss-0] org.jboss.netty.channel.SimpleChannelUpstreamHandler - EXCEPTION, please implement org.jboss.netty.handler.codec.http.HttpContentDecompressor.exceptionCaught() for proper handling.
java.net.ConnectException: Connection refused: /192.168.234.144:8100
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]