All - Recently our Druid cluster stopped responding for some time. The router was returning a HTTP 500 error. In the router logs, there was an exception (Catastrophic error) that both the brokers had gone offline just before the 500 error. In the broker logs, we found the below warnings around the time of error: Broker 1: WARN [main-SendThread(…..)] org.apache.zookeeper.ClientCnxn - Session 0x30000196c250005 for server ……, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_341] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_341] Broker 2: INFO [main-SendThread(….)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 23602ms for sessionid 0x2000017bd3f0002, closing socket connection and attempting reconnect Also, Broker 1 went OOM some time later. Error: Terminating due to java.lang.OutOfMemoryError: Java heap space Do the connection warnings indicate a n/w issue b/w the broker and zk? When would the broker go OOM? Heap space setting is 12GB. In addition, we see exceptions like the ones below in the broker logs. Exception 1: ERROR [qtp246535285-149] org.apache.druid.sql.avatica.DruidMeta - null org.apache.calcite.avatica.NoSuchConnectionException: null at org.apache.druid.sql.avatica.DruidMeta.getDruidConnection(DruidMeta.java:841) ~[druid-sql-24.0.2.jar:24.0.2] at org.apache.druid.sql.avatica.DruidMeta.createStatement(DruidMeta.java:239) ~[druid-sql-24.0.2.jar:24.0.2] Exception 2: ERROR [qtp246535285-130] org.apache.druid.sql.avatica.DruidMeta - Sequence iterator timed out org.apache.druid.query.QueryTimeoutException: Sequence iterator timed out at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$1$1.hasNext(ParallelMergeCombiningSequence.java:190) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.BaseSequence$1.next(BaseSequence.java:115) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.WrappingYielder$1.get(WrappingYielder.java:53) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.WrappingYielder$1.get(WrappingYielder.java:49) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-24.0.2.jar:24.0.2] Do these indicate any connectivity issue with the metadata DB? Does the broker connect directly to the metadata DB? Any inputs on diagnosing this would be really helpful. Thanks, AR.

the

OutOfMemoryError: Java heap space

is likely the root cause here; it means the Broker ran out of memory, and that likely caused all the other problems you saw

Thank you Gian. Router lost connection to both brokers about an hour before the brokers went down. The only warnings/errors in the broker logs were related to the ZK connection. So we were wondering if there was a n/w issue which disconnected the brokers from the router. About an hour later: Broker 1 shut down and restarted with no exception in the logs. It seems to have gracefully shut down for no apparent reason. 10 min after that Broker 2 went down with OOM and restarted. First time we are facing this issue. We will bump up the heap to 16GB and see if it recurs. What about the other 2 exceptions. We see them a few times every week in the logs. Exception 1: ERROR [qtp246535285-149] org.apache.druid.sql.avatica.DruidMeta - null org.apache.calcite.avatica.NoSuchConnectionException: null at org.apache.druid.sql.avatica.DruidMeta.getDruidConnection(DruidMeta.java:841) ~[druid-sql-24.0.2.jar:24.0.2] Exception 2: ERROR [qtp246535285-130] org.apache.druid.sql.avatica.DruidMeta - Sequence iterator timed out org.apache.druid.query.QueryTimeoutException: Sequence iterator timed out at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$1$1 Are these also indicating some transient n/w issue? Thanks, AR.

i am not 100% sure from just the logs, but those look likely to be transient problems to me