This message was deleted.

The context of the question is that we are seeing historicals are holding on to the segments past their retention durations, causing the historical boxes to go out of space to load new segments Our

Max segments to move

is set to 500, while

Balancer compute threads

are 7.

it runs on its own once a minute: there isn't a way to force it, but that's pretty often

To add, any chance you are using tiering in historicals? It also might be worth increasing the replication throttle limit setting on the coordinator. See - https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1656700175994759?thread_ts=1656633981.766599&cid=C0309C9L90D This is relevant especially if you are seeing log messages like below in your coordinator log.

<http://log.info|log.info>("Loading in progress, skipping drop until loading is complete");

thank you @Samarth Jain! Just go through the other conversation that you mentioned above. One concern is will increase

replicationThrottleLimit

affect the segment loading speed?

replicationThrottleLimit

is basically controlling number of segments that can replicated in one round. There is some weirdness in Druid where it doesn't execute the drop rule if replication is throttled. Do you see lots of "skipping drop" log lines in coordinator log?

yeah, i can see these log lines

yeah, very similar symptoms. In our case the hot tier was running out of space since the coordinator was never able to issue drop requests to remove the data from the historicals. Two things that we did that helped resolve the issue for us: 1. Increase

replicationThrottleLimit

2. Run two or more coordinators and bounce coordinators in a staggered fashion using a cron script. I would try with 1 first and see if helps resolve your issue. How many segments are there on your cluster?

yeah, i will try these options. i am not sure the exact number of segments, but i think it is more than 1m segments

Loading in progress, skipping drop until loading is complete

could occur upon addition of a new tier which is underrpelicated as Samarth has said

Yep, our cluster has 300k+ segments with tiering and realtime ingestion enabled and the above setup has helped. We have run into cases where the coordinator was just busy trying to move data to cold tier and not handoff any segments from the peons for a long time. So we had to resort to bouncing the coordinators in a staggered fashion otherwise handoff would just timeout and the tasks would fail and retry ingestion. I. short, bad news.

Is there an issue with balancing or are tiers getting overreplicated? If it's the former, reducing maxSegmentsToMove temporarily could prevent further worsening of the tiers

I meant the latter: overreplication on other tiers

Thank you for pointing to the github issue, @Amatya Avadhanula . I am glad to see focus on improving/fixing various issues in Druid coordinator.