This message was deleted.
# troubleshootings
Slackbot
06/08/2023, 5:21 AMThis message was deleted.
t
Tejas Parbat
06/08/2023, 6:14 AMDo you see any errors in coordinator logs, coordinator will instruct historical to drop/load segments.
Tejas Parbat
06/08/2023, 6:15 AMAlso what historical logs says
j
jp
06/08/2023, 6:20 AMI see the following log from coordinator
Copy code
2023-06-08T06:17:33,088 WARN org.apache.druid.server.coordinator.rules.LoadRule - No available [_default_tier] servers or node capacity to assign segment [0_base_2023-06-07T20:00:00.000Z_2023-06-07T21:00:00.000Z_2023-06-07T20:00:01.729Z]! Current replication: [[_default_tier:1/2]]
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.MarkAsUnusedOvershadowedSegments - Skipping MarkAsUnused as there are no overshadowed segments.
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Assigned 0 segments among 1 servers
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Dropped 0 segments among 1 servers
2023-06-08T06:17:33,089 WARN org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: insufficient active servers. Cannot balance.
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues:
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[10.233.92.61:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 343 served, 0 bytes queued, 10,442,387,194 bytes served.jp
06/08/2023, 6:21 AMand from historical , logs the below
Copy code
2023-06-08T06:20:17,803 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1462/23769][0_base_2023-06-07T22:00:00.000Z_2023-06-07T23:00:00.000Z_2023-06-07T22:00:01.153Z_11]
2023-06-08T06:20:18,280 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1463/23769][0_base_2023-06-02T19:00:00.000Z_2023-06-02T20:00:00.000Z_2023-06-02T19:00:00.714Z_144]
2023-06-08T06:20:18,364 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1464/23769][0_base_2023-06-01T09:00:00.000Z_2023-06-01T10:00:00.000Z_2023-06-01T09:00:00.504Z]t
Tejas Parbat
06/08/2023, 6:24 AMDid your retention rule changes saved in druid console , try bouncing leader coordinator if chnages didnt saved
j
jp
06/08/2023, 6:28 AMYes I checked retention rule changed and segments reduced. What do you mean "try bounding leader coordinator"?
t
Tejas Parbat
06/08/2023, 6:30 AMi was saying if changes are not taking effect then we can restart coordinator
j
jp
06/08/2023, 6:37 AMokay, I see. I've already tried restarting coordinator but still same.
jp
06/08/2023, 6:49 AMI've only set dropBeforeByPeriod, but if no load retention are set will affect this?
jp
06/08/2023, 7:17 AMI tried below but still same
1. set load retention to 1D and save
2. restart coordinator
3. restart historical
4. still trying to load 20k
k
Kyle Hoondert
06/08/2023, 8:00 AMYou might want to try cleaning the segmentCache volume. This will be re-populated by the coordinator. At the moment it might be loading the segments it has in the volume already
👍 3
j
jp
06/08/2023, 9:29 AM@Kyle Hoondert I cleaned the segmentcache volume as you said. It's now working properly. Thank you
😎 1
👍 1
jp
06/09/2023, 12:47 AM@Kyle Hoondert I thought when historical starts, it would talk to coordinator or zookeeper to get metadata for current cluster state. But seems like druid uses local cache, or local metadata first. Is this right?
k
Kyle Hoondert
06/09/2023, 7:18 AMYes, the historical first looks at it’s local segment cache and attempts to make this available. When this is complete, it communicates with the coordinator to add/remove segments served
j
jp
06/10/2023, 1:58 AMThank you!!