This message was deleted.
# troubleshooting
s
This message was deleted.
t
Do you see any errors in coordinator logs, coordinator will instruct historical to drop/load segments.
Also what historical logs says
j
I see the following log from coordinator
Copy code
2023-06-08T06:17:33,088 WARN org.apache.druid.server.coordinator.rules.LoadRule - No available [_default_tier] servers or node capacity to assign segment [0_base_2023-06-07T20:00:00.000Z_2023-06-07T21:00:00.000Z_2023-06-07T20:00:01.729Z]! Current replication: [[_default_tier:1/2]]
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.MarkAsUnusedOvershadowedSegments - Skipping MarkAsUnused as there are no overshadowed segments.
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Assigned 0 segments among 1 servers
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Dropped 0 segments among 1 servers
2023-06-08T06:17:33,089 WARN org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: insufficient active servers. Cannot balance.
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues:
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[10.233.92.61:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 343 served, 0 bytes queued, 10,442,387,194 bytes served.
and from historical , logs the below
Copy code
2023-06-08T06:20:17,803 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1462/23769][0_base_2023-06-07T22:00:00.000Z_2023-06-07T23:00:00.000Z_2023-06-07T22:00:01.153Z_11]
2023-06-08T06:20:18,280 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1463/23769][0_base_2023-06-02T19:00:00.000Z_2023-06-02T20:00:00.000Z_2023-06-02T19:00:00.714Z_144]
2023-06-08T06:20:18,364 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1464/23769][0_base_2023-06-01T09:00:00.000Z_2023-06-01T10:00:00.000Z_2023-06-01T09:00:00.504Z]
t
Did your retention rule changes saved in druid console , try bouncing leader coordinator if chnages didnt saved
j
Yes I checked retention rule changed and segments reduced. What do you mean "try bounding leader coordinator"?
t
i was saying if changes are not taking effect then we can restart coordinator
j
okay, I see. I've already tried restarting coordinator but still same.
I've only set dropBeforeByPeriod, but if no load retention are set will affect this?
I tried below but still same 1. set load retention to 1D and save 2. restart coordinator 3. restart historical 4. still trying to load 20k
k
You might want to try cleaning the segmentCache volume. This will be re-populated by the coordinator. At the moment it might be loading the segments it has in the volume already
πŸ‘ 3
j
@Kyle Hoondert I cleaned the segmentcache volume as you said. It's now working properly. Thank you
😎 1
πŸ‘ 1
@Kyle Hoondert I thought when historical starts, it would talk to coordinator or zookeeper to get metadata for current cluster state. But seems like druid uses local cache, or local metadata first. Is this right?
k
Yes, the historical first looks at it’s local segment cache and attempts to make this available. When this is complete, it communicates with the coordinator to add/remove segments served
j
Thank you!!