Hi All, I doing the druid cluster setup recently. My druid cluster setup up and running file after 2 hours my broker and router went offline below the logs file only in broker.log. other are working fine(historical, middle and coordinator) 2024-05-13T034511,768 INFO [Thread-73] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER] 2024-05-13T034511,777 INFO [Thread-73] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@670ae31f{HTTP/1.1, (http/1.1)}{0.0.0.0:8082} 2024-05-13T034511,778 INFO [Thread-73] org.eclipse.jetty.server.session - node0 Stopped scavenging 2024-05-13T034511,781 INFO [Thread-73] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@46d51d5e{/,null,STOPPED} 2024-05-13T034511,792 INFO [Thread-73] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL] 2024-05-13T034511,792 INFO [Thread-73] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='10.0.18.133:8082', hostAndPort='10.0.18.133:8082', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=broker, priority=0}] 2024-05-13T034511,793 INFO [Thread-73] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping... 2024-05-13T034511,793 INFO [Thread-73] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped. 2024-05-13T034511,793 INFO [Thread-73] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView is stopping. 2024-05-13T034511,793 INFO [Thread-73] org.apache.druid.sql.calcite.schema.MetadataSegmentView - MetadataSegmentView is stopped. 2024-05-13T034511,793 INFO [DruidSchema-Cache-0] org.apache.druid.segment.metadata.AbstractSegmentMetadataCache - Metadata refresh stopped. 2024-05-13T034511,794 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore. 2024-05-13T034511,795 INFO [Thread-73] org.apache.druid.guice.LifecycleForkJoinPoolProvider - Shutting down ForkJoinPool [org.apache.druid.guice.LifecycleForkJoinPoolProvider@1ca4e30d] 2024-05-13T034511,800 INFO [Thread-73] org.apache.druid.client.HttpServerInventoryView - Stopping executor[FilteredHttpServerInventoryView]. 2024-05-13T034511,800 INFO [Thread-73] org.apache.druid.client.HttpServerInventoryView - Stopped executor[FilteredHttpServerInventoryView]. 2024-05-13T034511,812 ERROR [HttpClient-Netty-Worker-2] com.google.common.util.concurrent.AbstractFuture - RuntimeException while executing runnable CallbackListener{org.apache.druid.server.coordination.ChangeRequestHttpSyncer$1@5bf243f6} with executor java.util.concurrent.ScheduledThreadPoolExecutor@2968e49f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 70] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@108292b8 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2968e49f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 70] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_412] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_412] at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_412] at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_412] at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622) ~[?:1.8.0_412] at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286) ~[guava-32.0.1-jre.jar:?] at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055) ~[guava-32.0.1-jre.jar:?] at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807) ~[guava-32.0.1-jre.jar:?] at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55) ~[guava-32.0.1-jre.jar:?] at org.apache.druid.java.util.http.client.NettyHttpClient$1.handleExceptionAndCloseChannel(NettyHttpClient.java:365) ~[druid-processing-29.0.1.jar:29.0.1] at org.apache.druid.java.util.http.client.NettyHttpClient$1.channelDisconnected(NettyHttpClient.java:334) ~[druid-processing-29.0.1.jar:29.0.1] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:570) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:79) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:325) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_412] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_412] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_412] 2024-05-13T034511,820 INFO [HttpClient-Netty-Worker-2] org.apache.druid.java.util.http.client.pool.ResourcePool - giveBack called after being closed. key[http://10.0.15.202:8083] 2024-05-13T034511,851 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting 2024-05-13T034511,963 INFO [Thread-73] org.apache.zookeeper.ZooKeeper - Session: 0x10015d209610014 closed 2024-05-13T034511,963 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10015d209610014 2024-05-13T034511,964 INFO [Thread-73] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT] 2024-05-13T050056,136 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 6.2.5.Final 2024-05-13T050057,098 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-hdfs-storage], jars: wildfly-openssl-1.1.3.Final.jar, slf4j-api-1.7.36.jar, hadoop-aws-3.3.6.jar, commons-logging-1.1.1.jar, httpcore-4.4.11.jar, jackson-dataformat-cbor-2.12.7.jar, hadoop-client-runtime-3.3.6.jar, aws-java-sdk-core-1.12.638.jar, commons-codec-1.16.0.jar, hadoop-client-api-3.3.6.jar, snappy-java-1.1.10.4.jar, jmespath-java-1.12.638.jar, druid-hdfs-storage-29.0.1.jar, aws-java-sdk-s3-1.12.638.jar, jsr305-2.0.1.jar, httpclient-4.5.13.jar, aws-java-sdk-kms-1.12.638.jar, joda-time-2.12.5.jar 2024-05-13T050057,118 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-kafka-indexing-service], jars: lz4-java

typically druid servers don't go offline on their own, usually something kills or terminates them. do you see any hints on your system about what might have done that? sometimes it's the linux oom-killer, which you could see in

dmesg

thanks @Gian Merlino, let me check in that perspective.

Router is just a proxy for the other processes. It doesn't have a large memory footprint either. As Gian suggested, do check if there is some external process that is killing it.