This message was deleted.

I don't know of a calculation, unfortunately. I don't think segment sizes matter, just the number of segments. I doubt it would be perfectly linear, but maybe close? That said, I've seen it struggle when you get near 1M segments, but maybe that's just me.

I kinda assumed segment size wouldn’t matter but inevitably at some point the discussion would turn to “why so many segments”/compaction etc etc so thought I’d state that up front. 😂

I have heard of people with that many. Maybe @Didip Kerabat? (Did you have > 1M segments Didip? If so, how do you tune coordinator heap?)

The maximum number of segments you can have, with the current Coordinator architecture, is 5M. Beyond that, the entire cluster will be very unhappy. It doesn’t matter how large of heap I gave the leader. This is because coordinator caches all of the segment rows in RAM. Ideally, those rows are indexed properly inside PG and thus Coordinator does not need to cache anything on-heap. Or maybe the cache can be offloaded to Redis? This is why we have 1 compaction job per table: To aggressively reduce number of segment files.

Interesting take. In your experience, what happens to the cluster at that point to make it unhappy? Coordinator duty cycles taking too long?

correct. Beyond 5M segment files, everything that Coordinator needs to do, that requires access to that list of segment files, takes forever. And accordingly, the Coordinator GC graph goes wild (specifically the young objects).

We’ve always known the coordinator is the current limitation on properly scaling Druid. What heap size do you configure for 5 million segments?

Our largest cluster has 64GB RAM for Coordinator. The -Xmx setting is 64GB - 20% * 64GB. This puts a damper on our dream to build a true mega Druid cluster. Because of this, we had to provision separate Druid clusters per large internal customers.

Ah interesting. We’ve had test clusters with similar heap sizes and more segments (128GB RAM and ~7 million segments). The trade off we had to make was on real time data. We know the coordinator cycles take time which means our ingest handoff is delayed too. We’re playing with scaling it even higher and are running into memory issues at the coordinator (as expected). There’s a theory that scaling the coordinator memory vertically will allow it to manage more segments (at the cost of duty cycles) but wondered if this had been tested in practice and whether there was an actual method of working this out (much like you could in the old Hadoop name node heap size (roughly 1GB per million blocks)).

We also encountered issues with higher segment numbers (over 500k - 1Million), as segment balancing algorithms, etc. seem to eat up all ressources (CPU and memory). Regarding heap I can recommend just looking at the JVM memory graphs. If the saw tooth (memory getting freed by GC and then slowly fills up again) is very large (50% of your memory "height") then you are wasting memory. If the sawtooth is medium (e.g. about 30% of the memory), then it's perfect. If the sawtooth is small (smaller than 10% of total memory) then you should give it more memory. We don't have direct JVM memory (e.g. via jmx exporter), but use Imply Clarity for memory monitoring. That said, for our setup 16g heap was enough for a stable operation of about 300 - 500k segments. I cannot overstate the importance of compactions and checking that segment sizes are around 300-700MB size (or if you have very small rows, then max. 5 Mio. rows). We had lots of scattered segments due to late arrival of data and this can blow up your segment count ten or 100 times. This also affects query performance greatly. Compactions are as important as ingestions for cluster health.

The “high number of segment files” problem is so bad that we are willing to take a small performance hit by enlarging the segment size to 1GB-2GB. It’s way better than having the entire cluster failed.