Hi All, I wanted to check if there is a way to clean up only the older segment versions from metadata and deep storage, while retaining the latest versions safely in deep storage. We have a use case where we want to maintain retention in Druid as per policy. However, since we are running frequent compactions, we do not want to retain older segment versions unnecessarily. I came across the following properties: •

druid.coordinator.kill.on

•

druid.coordinator.kill.period

•

druid.coordinator.kill.durationToRetain

From what I understand, these properties delete all unused segments after a certain period, but not specifically older versions if newer versions exist. Is there a way to configure Druid to only delete older segment versions while keeping the latest one for each interval?

yes, that's correct, automated kill tasks currently delete all versions of unused segments in an interval. However, you can submit manual kill tasks with specific

versions

in the payload and those will selectively delete only the specified version(s)

> I wanted to check if there is a way to clean up only the older segment versions from metadata and deep storage, while retaining the latest versions safely in deep storage This is how kill tasks work exactly. It should not require any tweaks.

You can configure "cold tier" for these time intervals by setting retention to "LoadForever" with zero (0) replicas ... that will leave the most recent version of the segments per time interval as used, and the older ones (e.g. compaction discards) will be marked as unused and can be cleaned up with the kill tasks. NOTE -- if you are starting with segments all marked as unused, then when you add the new cold tier rule you must then re-mark those intervals as used, so the coordinator can properly filter out the current version from the prior version of the segments, and leave only the current ones marked as used. If you do not do this second step then all segments in the interval may still be marked a unused. (ref: https://druid.apache.org/docs/latest/operations/rule-configuration#reload-dropped-data)