This message was deleted.
# troubleshooting
s
This message was deleted.
v
are you talking about disk usage or historical fill level? Are all the historicals on identical hardware?
b
@Vadim Yes, all are identical have the same hardware. When i look at output of /druid/coordinator/v1/servers?simple maxSize is ~6TB for all histroicals, but current size is much higher on few historicals, ie they are not balanced
Copy code
"tier": "_default_tier",
    "type": "historical",
    "priority": 0,
    "currSize": 6247982185627,
    "maxSize": 6248103673856

    "tier": "_default_tier",
    "type": "historical",
    "priority": 0,
    "currSize": 3927430171420,
    "maxSize": 6248103673856
Copy code
curl -sL <http://druid-coordinator:8080/druid/coordinator/v1/servers?full> | jq '.[]|{host:.host, segments: [.segments|to_entries[] | .key]}' | jq -c '[.host, (.segments|length)]'
["druidHistoricalBroker-6:8100",1438]
["druidHistoricalBroker-15:8100",18128]
["druidHistoricalBroker-9:8100",18988]
["druidHistoricalBroker-14:8100",18755]
["druidHistoricalBroker-8:8100",12589]
["druidHistoricalBroker-3:8100",17947]
["druidHistoricalBroker-12:8100",18283]
["druidHistoricalBroker-10:8100",17882]
["druidHistoricalBroker-13:8100",28791]
["druidHistoricalBroker-5:8100",29536]
["druidHistoricalBroker-1:8100",17969]
["druidHistoricalBroker-11:8100",19848]
["druidHistoricalBroker-2:8100",29918]
["druidHistoricalBroker-4:8100",18312]
["druidHistoricalBroker-7:8100",20744]
["druidHistoricalBroker-16:8100",19488]
b
@Bastian You may try setting :
druid.coordinator.balancer.strategy
Use
diskNormalized
to distribute segments among processes so that the disks fill up uniformly and use
random
to randomly pick processes to distribute segments. https://druid.apache.org/docs//0.15.1-incubating/configuration/index.html
b
@Bibek Sahoo i changed this setting and restarted coordinators waiting 8 hours nothing changed.
b
May be try to restart the master node and check the log for further information.