https://linen.dev logo
Join Slack
Powered by
Hi All, In the IOConfig section of the Compaction ...
# general
a
Hi All, In the IOConfig section of the Compaction spec, what happens if the flag "dropExisting" is set to the default value of "false"? Does it create duplicate segments for the same time interval and marks the earlier ones as "unused"? If so, is it possible to revert back to the original set of segments? My understanding is that only one set of segments is available and active for an interval in Druid. It is possible for a segment to be overshadowed by another more recent segment if the latest segment is not "published" yet (loaded to the historical segment cache). But once the latest segment is published, the previous segment is removed from deep storage and metadata. Is this understanding incorrect? Thanks, AR.
a
Does it create duplicate segments for the same time interval and marks the earlier ones as "unused"? If so, is it possible to revert back to the original set of segments?
Yes. It is possible to revert to the original set of segments if you know the old version and interval
1. Mark all segments in the interval as unused. 2. Wait for the historical to drop these segments. This will mean that queries wouldn't have data from this interval temporarily 3. In your metadata store (mysql / postgresql), mark the desired set of segments (using the version and interval to identify them) as used by setting the column 
used = true
4. The coordinator will assign the previous set of segments that are the used segments for the version to historicals 5. The datasource will serve the older version once the historicals have loaded and announced these segments
a
Thank you Amatya. Is this always the case or is it specific to compaction? If we reindex the data (not compaction) for an interval - either from the same Druid table or from an external source, would the same thing happen? Will it also create duplicate segments and mark the old ones unused or would it delete the old ones in this case? Thanks, AR.
a
Yes, old segment will be retained until a kill task cleans them up
a
Thank you for clarifying my understanding. I thought only segments that were marked "unused" explicitly (manually) needed to be cleaned up by submitting kill tasks manually. But you are saying that any rewrite of data in Druid will only mark the previous version as "unused" but not delete it from deep storage & metadata. Thanks, AR.
👍 1
a
Segments generally have the following states: 1. Used and visible 2. Used and overshadowed 3. Unused 4. Deleted by kill tasks
Potentially 0. Allocated and uncommitted
So, yes. A segment that is currently used and visible may be overshadowed by a newer version. It can then be marked as unused by the coordinator Kill tasks delete the metadata entries and deep storage files finally. You can also add a buffer period between the time at which a segment is marked as unused and killed
a
Used and overshadowed --> This would be a transient state of a segment. Correct? Thank you for the detailed explanation. I believe you are referring to "druid.coordinator.kill.on" config setting. This should clean up all old segments - reindexed & compacted. Correct? Thanks, AR.
a
Used and overshadowed --> This would be a transient state of a segment. Correct?
Yes.
"druid.coordinator.kill.on" config setting. This should clean up all old segments - reindexed & compacted.
Yes.
druid.coordinator.kill.bufferPeriod
is the buffer period that I was referring to. Your kill duty frequency, the number of kill slots etc also determine the rate at which unused segments are cleaned up.
a
Thank you Amatya.
2 Views
Open in Slack
PreviousNext
Powered by