This message was deleted.

Hello @Caner Kürtür, the drop policy is run automatically. 1. Once segments have been dropped from the clusters, they are marked as

unused

in the metadata store (sort of a logical deletion). 2. To permanently delete unused segments, there is a need to execute kill tasks https://druid.apache.org/docs/latest/data-management/delete.html#permanently-kill-task . 3. Kill Tasks will completely delete unused segments from both deep storage and metadata store. 4. Kill Tasks can be executed manually or scheduled externally (e.g through Airflow).

so do we have to apply this everytime when we want to drop? isn't it like: when we set a retention policy for a data source, it is going to apply these progress for us in automated way whenever it sees some segments older than 2 months?

The drop is automatic. If you set the retention policy for 2 months, then only the most recent 2 months of data will remain loaded on your data nodes.

because, I saw in the UI that I can mark segments manually myself and kill them but I would like to automate this, I dont want to keep data older than 2 months

However, dropping the data from the data nodes does not delete it from deep storage. That's what the kill task does.

so it is going to automate only the "marking as unused" part for the outdated segments right?

You can automate it in Druid.

Thank you so much Anil! so helpful! appreciated 🙂 I will definitely check the links out that you have sent. It is great to hear that we can also automate kill task via the Coordinator configs

My pleasure 🙂

One more question regarding that maybe: I just saw "Coordinator dynamic config" in the Druid UI. There is "Kill data source whitelist" section we can set up, is it also why it is used for? like I mean can we automate kill task setting it in UI? 🙂

My read of the doc is that the configs we are interested in are part of the static configuration rather than the dynamic configuration. I'm pretty certain that making changes to the dynamic configuration won't accomplish what we want.

I see, alright 🙂 and how can I reach out to coordinator/runtime.properties file actually and change the settings? Our Druid is Imply hosted. Just not sure how we can set the settings, should I reach out to them for changes?

@Caner Kürtür, if your Druid is Imply Hosted, you can do it this way: https://support.imply.io/hc/en-us/articles/360035308573-Deep-storage-data-segment-deletion-using-coordinator-kill

Thanks a lot! much appreciated, I will check it out 🙂