Hi All, Is there any guideline for ZK memory sizing? We were running our ZK with 128MB heap size. The ZK is only used by Druid. Our PROD has ~275K segments and when we dropped a table with ~26K segments, it caused the ZK to go OOM. We restarted the ZK with 256MB and it is running fine again. Is there a way we can estimate the heap size of ZK ? From the docs, the significant data that might consume ZK memory is: • Served segments - as each historical will create a ephemeral node for each segment that it is serving. So as number of segments increases, the number of such nodes will increase. • Segment Load/Drop announcements from Coordinator to Historical - These are temporary but there could be many if there are frequent load jobs. In our case, number of segments is ~275K and increasing. We also have frequent REPLACE jobs running to overwrite existing data as updates are not possible in Druid. These jobs run at intervals of 1 hr - 1 day and there are plans to add weekly jobs to overwrite full history of some of the tables. So it is important to understand the effect of these on ZK and size it accordingly. Any pointers on this will be very helpful. Thanks, AR.

that's too little memory for ZK I only could keep ZK stable with 2GB with ~ 20k segments

Hi Renato, 2GB is a lot of memory for ZK. Why would it need so much? We have been running Druid cluster for about 8 months and data has increased from 0 to ~275K segments. After increasing ZK JVM size to 256MB, we can see that ~125 MB is free. The resident size of the process is ~430MB. Thanks, AR.

I set so it did not crash zk-persistent-0 234m 633Mi zk-persistent-1 220m 1051Mi zk-persistent-2 85m 1035Mi In the page it sasys Required Software ZooKeeper runs in Java, release 1.8 or greater (JDK 8 LTS, JDK 11 LTS, JDK 12 - Java 9 and 10 are not supported). It runs as an ensemble of ZooKeeper servers. Three ZooKeeper servers is the minimum recommended size for an ensemble, and we also recommend that they run on separate machines. At Yahoo!, ZooKeeper is usually deployed on dedicated RHEL boxes, with dual-core processors, 2GB of RAM, and 80GB IDE hard drives. https://zookeeper.apache.org/doc/r3.8.1/zookeeperAdmin.html#:~:text=Required%20Software,-ZooKeeper%20runs%20in&text=Three%20ZooKeeper%20servers%20is%20the,and%2080GB%20IDE%20hard%20drives.

I used the ZK jvm config from the config available in the Druid tar ball. It is set to 128MB. Are you using a common ZK that is used by apps other than Druid? We are using a dedicated ZK cluster for Druid and it works fine with 256MB of RAM for ~250K segments. I guess the issue comes when large number of segments are affected in a single transaction - all segments in a table are marked as "unused". This will cause the co-ordinator to push the list of segments (double the count including replication) that need to be dropped by each historical. I am not sure in which format this list is but it could be huge if it is text/json. This is what caused the ZK to crash in our case when it was running with 128MB. Regards, AR.