sorry, I missed responding for a while. 2 million is quite a lot of segments. it is possible to tune the system to work well with 2 million segments, but before going down that route, I'd want to first confirm if you actually need 2 million segments. what is their average size, in number of rows? generally we look to have 3ā5 million rows per segment on average. if yours are smaller than that, then I would recommend using compaction to reduce the number of segments.
on the other hand, if yours are about that size, that means you have a trillions of rows dataset. congrats, that's really big data š. in this case i have a few things to consider:
⢠what version of Druid are you using? with more recent versions, ZooKeeper is used for much less- pretty much just leader election and service discovery. There should be no ZooKeeper scalability issues with large segment counts in the more recent versions
⢠for the Broker and Overlord both, I'm curious what kinds of crashes are you seeing? You mentioned having 90GB+ for Brokers- even at 2 million segments that should be more than enough memory. I wonder if you are seeing OutOfMemoryErrors or something else