Hi all,
We've encountered an issue where our deep storage had a temporary issue and couldn't provide one of our historicals with a segment. The historical logged a "Failed to load segment for dataSource" error and went about as normal.
Unfortunately, druid still believes that segment is loaded on that historical (as shown by "select * from sys.segments where segment_id = '<segment_id>'".
When running queries that hit this segment, we're now seeing "org.apache.druid.segment.SegmentMissingException" errors. How do we get out of this situation?
Restart all historicals should force the to read all segments from disk and update the coordinator right? Does the coordinator ever do a periodic check of historicals to ensure they're hosting all the segments it believes they are?