This message was deleted.

I don't have a figure handy for what to expect, although 5 hours definitely seems like a long time, and longer than I'd expect…

The next time we bounce it, I'll try to grab a flame graph and share...

ok 🤞

I'm currently unable to send the full flame graph, but looking at things, we have two threads using any meaningful resources (CoordinatorServerView-0 and CachingCostBalancerStrategy-executor) Both threads are seemingly maxing out a CPU which appears to be the bottleneck (the pod itself has lots of unused CPU). Does the CachingCost balancer scale well? What are other large multi million segment clusters using for their balancer strategy and what are the startup times like?

I'm wondering which version you're using? We recently (I forget the release; sometime in the last couple majors IIRC) made a bunch of speed improvements to the balancing logic geared towards clusters with ~millions of segments Another thing: I'm wondering if you're setting

balancerComputeThreads

in your Coordinator dynamic config? You mention "maxing out a CPU" — for a cluster with ~millions of segments I'd expect balancing to be maxing out more CPUs 🙂 For a cluster that size, reasonable value might be 32 on a machine with as many CPUs Wondering if @kfaraz or @Amatya Avadhanula have any other pointers— they've been involved in the Coordinator scaling efforts

We're using 25.0.0. We provide 40 cores to the coordinator pod, and the balancerComputeThreads is set to 16. For the "maxing out a CPU" statement. It's very possible I'm misinterpreting the output of ttop (part of the SJK). Running ttop, these threads never use more than 100% CPU (although independently each thread will use > 97% which is why I assumed it wasn't an overall percentage and was akin to OS level CPU metrics (100% = 1 core)). Memory seems to be a problem too as our pod currently needs a 200GB resource limit as even with that heap size, we're still being OOMKilled.

Hi Could you please try the following coordinator dynamic configs? 1.

Use Round robin segment assignment - true

[This will assign segments in a round robin fashion and let the cost strategy decide balancing happen later] 2.

Use batched segment sampler - true

[For efficient sampling of segments for balancing] 3.

Max segments to move - 500 to 1000

[For faster balancing since round-robin doesn't assign in the most optimal fasion] 4.

Max segments in node loading queue - 100 - 1000

[This limits the assignments / moves of segments in a given cycle per historical] 5.

Replication throttle limit - 2000

[This limits the number of non primary segment replicas assigned per cycle, you could choose a higher value]

The dynamic config you suggested was pretty much what we run already: 1. Round robin was set to false (now true) 2. Batched segment sampler already set to true 3. Max segments to move was 500 4. Max segments in node loading queue was 500 5. replication throttle limit was 500 I've switched to the cost balancer and will let you know a comparative time when it comes up 👍

are you able to share the capture you took (the stcap or html file) or a screenshot showing the above ☝️ ? i'm wondering what the call stack looks like for those

isOvershadowed

calls

It's on an airgapped system so not easily.... com.google.common.collect.iterators$5.hasNext org.apache.druid.timeline.VersionedIntervalTimeline.isOvershadowed org.apache.druid.timeline.SegmentTimeline.isOverShadowed org.apache.druid.client.DataSourcesSnapshot.determineOvershadowedSegments org.apache.druid.client.DataSourcesSnapshot.fromUsedSegments org.apache.druid.metadata.SqlSegmentsMetadataManager.doPoll org.apache.druid.metadata.SqlSegmentsMetadataManager.poll org.apache.druid.metadata.SqlSegmentsMetadataManager.lambda$createPollTaskForStartOrder$0

We did some optimizations there. In particular, we reduced the number of times we call

isOvershadowed

from

MarkAsUnusedOvershadowedSegments

duty. The

isOvershadowed

method call itself has not been optimized.

Our densest timechunk would be around 20,000 segments (we're experimenting with ingest and would expect this to reduce considerably)

Hmm, that is probably what's causing this.

got it. I wonder if this is indeed related

@Nick M, if you run compaction for those densely packed intervals, you will definitely get much better performance.

we have done a bunch of optimization work recently geared at large total numbers of segments, but not as much geared at very dense time chunks (to me "very dense" is anything 10,000+)

Yeah. compaction is next on our list. We've just about scaled ingest (the tuning of which has caused all of these smaller segments). If I could get the coordinator up I'd manually compact the bigger timechunks but I'm seemingly in a loop where the coordinator keeps running out of memory on startup

@kfaraz do you know if there's an open issue or any investigation that's been done into startup & balancing times with very dense time chunks?

No, I don't think we have done any work with very dense time chunks in the recent past. The solution in such cases has almost always been to compact such intervals, as the segments always turned out to be very small indeed, and thus compaction helped.

yeah i'm wondering if we could optimize this, so when people find themselves in this situation, it's easier to actually get the compaction done— seems like it's tough to get it done since the Coordinator is not super responsive