https://linen.dev logo
Join Slack
Powered by
Is there way to force rebalance the segments? I ch...
# general
y
Is there way to force rebalance the segments? I changed strategy to 
druid.coordinator.balancer.strategy=diskNormalized
nothing changes.
maxSegmentsToMove 100 ->10000
a
Disk normalised is quite slow iirc. Which version of druid are you using?
y
I am using 30.0
a
Have you tried smart segment loading?
y
it is enabled.
a
If it is not working as expected, what is the segment count on your cluster?
y
Copy code
select COUNT(*)
from sys."segments"
where "is_active" =1
result 130,401
a
That's not a lot. How long has it been since you added new historicals?
y
its been a day for now. but strategy was cache. I changed it to diskNormalized to 6 hours ago.
a
What do the loadqueues look like?
y
You meant content of the queueu?
“maxSegmentsInNodeLoadingQueue”: 10000, this is how I set.
a
Yes. How many segments are being loaded / dropped for each historical
The information on the right side of each historical in the services page
y
it says empty
a
Strange. Could you please try restarting your coordinators? Any logs in your coordinators before you restart them would be helpful too
y
Is it because of compaction? I set to run compaction every minute. In coordinator log, I sometimes see
Copy code
Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Duplicate entry 'datasource' for key 'druid_pendingSegments.PRIMARY'
but I believe this is due to kafka ingestion running 3 replica for single partition.
a
Compaction should not affect balancing afaik, nor should active ingestion
✅ 1
Balancing affects only segments loaded on historicals
y
I do not see any other log, what log config should I set to check?
a
Something like:
Copy code
2024-07-23T09:16:14,003 INFO [Coordinator-Exec-HistoricalManagementDuties-0] org.apache.druid.server.coordinator.duty.BalanceSegments - Computed maxSegmentsToMove[20] for total [20] segments on [1] historicals.
2024-07-23T09:16:14,003 INFO [Coordinator-Exec-HistoricalManagementDuties-0] org.apache.druid.server.coordinator.duty.BalanceSegments - Balancing segments in tiers [[_default_tier]] with maxSegmentsToMove[20] and maxLifetime[60].
y
I will enable log in log4j and will comeback. thank you for your help.
👍 1
It turns out that I mounted wrong nodeConfigPath on K8s. in k8s operator.
Copy code
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/master/coordinator"
change to
Copy code
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/master/coordinator-overlord"
since I am using coordinator with overlord. That is why I could not see any logs nor configs not working properly. It is working fine now.
🙌 1
Open in Slack
PreviousNext
Powered by