I've been noticing disk filling up on my Druid Deep Storage host. After some investigation I've noticed that Druid seems to lose track of segments sometimes. I'm not really sure why but we did have an incident where disk reached 100% which could be a root cause. In any case, when Druid issues a kill for the time period, Deep Storage retains a bunch of files. I manually have to delete these files. Is there any function in Druid that can discover untracked segment files that exist in Deep Storage but not in the metadata DB?

I have noticed the same thing. It seems to happen if the cluster hits issues (like disk full) for extended periods such as things going bad overnight and having to fix them in the morning. I make it a normal part of my day to go through and look for files in deep storage that are older than what the retention rules allow for then manually delete them

I considered establishing a cron job. However, it's annoying to have to keep the retention policy in sync, since it can easily be changed in the Druid UI. A deeper issue which I discovered is when there is no retention policy or the policy is large (1 year): old segments left over from compaction might be left around. I recently manually cleaned up 500 GiB of old versions of segments that we will retain for 6 more months. I think it's possible that segments are pushed to deep storage during a task but the task subsequently fails for some reason (e.g. OOM). Either Druid lost track of the segment or the segment is simply never marked unused.