This message was deleted.

Coordinator dynamic config -

{
  "millisToWaitBeforeDeleting": 900000,
  "mergeBytesLimit": 524288000,
  "mergeSegmentsLimit": 100,
  "maxSegmentsToMove": 200,
  "percentOfSegmentsToConsiderPerMove": 100,
  "replicantLifetime": 15,
  "replicationThrottleLimit": 150,
  "balancerComputeThreads": 7,
  "emitBalancingStats": false,
  "killDataSourceWhitelist": [],
  "killAllDataSources": false,
  "killPendingSegmentsSkipList": [],
  "maxSegmentsInNodeLoadingQueue": 1000,
  "decommissioningNodes": [],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 0,
  "pauseCoordination": false
}

Can you check the coordinator logs to see why these segments are not getting loaded?We can click on the unavailable segments on the console which will show the segment id of them, then we can try to grep for those ids in the coordinator logs

If you are collecting error logs from Historicals, you may notice errors about segments failing to load, as well

Thanks a lot @Hemanth Patil, and @Gian Merlino for your pointers. We were able to identify the root cause of the issue which was - one historical box was not able to load segments because of disk issues. Taking that historical down stabilized the entire cluster, and all segments loaded correctly. Two follow up questions on this behavior - 1. We were not able to see the segments trying to be loaded on the historical from the coordinator endpoint,

druid/coordinator/v1/loadqueue

. That historical had 0 segments to load, and 0 segments to drop. Wondering why would this happen? 2. This historical is part of a low priority tier, but there are other tiers that are part of some highly critical revenue barring system which are must for our operations. How can we avoid one historical's misbehaviors from impacting other tiers? The coordinator got too busy in dealing with this historical, which caused the entire cluster to suffer. Thanks!

That historical had 0 segments to load, and 0 segments to drop. Wondering why would this happen?

☝️ just a guess, but it could be failing so fast that its queue was empty

just a guess, but it could be failing so fast that its queue was empty

Yes, this is possible.

How can we avoid one historical's misbehaviors from impacting other tiers?

The coordinator already handles each tier separately. Did you see any exceptions in the coordinator logs when this was happening?

The coordinator got too busy in dealing with this historical, which caused the entire cluster to suffer.

We are working to ensure that the coordinator doesn't keep trying to assign segments to unresponsive/failing historicals. But that said, this shouldn't have affected loading on other tiers.