Hi. My team is running an instance of Druid 24.0.2 and we are seeing some strange issues with Zookeeper. For context, we have ~75 Druid workers (50 MMs split between Kinesis realtime ingestion and SQL backfill, 2 coordinators, 16 historicals, 1 broker, 2 routers) all in Amazon ECS. We are running 3 Zookeeper nodes, 1 leader with 2 followers. We do realtime ingestion using Amazon Kinesis. When any one of the Zookeeper nodes goes down, the Druid cluster loses much of its state. Ingestion tasks fail and restart, the coordinator throws a number of errors, and query performance times out or p99 latency increases by orders of magnitude. My expectation with Zookeeper would be that the other two ZK nodes would be able to maintain state and there would be no service interruption. The Druid workers are passed all three Zookeeper hosts like this:

druid_zk_service_host: 1.nodes.zookeeper,2.nodes.zookeeper,3.nodes.zookeeper

We also notice that the Zookeeper leader needs to be node 3 for Druid to function. Leader re-election will happen when the leader goes offline, but Druid is largely non-functional unless the leader is 3. I have troubleshooted a number of possible causes, including disabling DNS caching in the JVM. Here are some logs from Zookeeper and coordinators when one of the ZK nodes goes offline:I did see a lot of this in the Zookeeper logs:

INFO  [QuorumConnectionThread-[myid=1]-3:QuorumCnxManager@481] - Have smaller server identifier, so dropping the connection:

And here is a Druid coordinator in the same time window:

WARN [Curator-PathChildrenCache-3] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[10.0.123.88:8091] announced a status for a task I didn't know about, adding to runningTasks: index_kinesis_timeseries-rollup_4cd6adcb74274f1_bfpgfhlk

ERROR [IndexTaskClient-impressions-hour-9] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Failed to get task runner because I'm not the leader!

I'm wondering if you all have encountered similar issues with Zookeeper, or have any suggestions on new avenues for troubleshooting. In my testing, restarting or modifying any of the Zookeeper nodes causes production impact. We would like to be able to upgrade the underlying EC2 instances running Zookeeper without service interruption, and also have some insurance in case one of those nodes becomes unresponsive due to other issues.