This message was deleted.

Just to make sure, have you checked the docs here? https://druid.apache.org/docs/latest/operations/rule-configuration.html

yes I have gone the doc. @Luiz Augusto

My understanding is "BeforeBy" means anything older than the timeframe specified, and "By" means anything newer than the timeframe specified. "Drop" rules have "BeforeBy" and "By" variations, whereas "Load" only has a "By" variation. The rules are evaluated in the order listed in the ruleset. Assuming you have listed each ruleset on one line and comma-separated the rules in their order of evaluation, then I interpret your first two rulesets as: 1. Drop anything order than 580 days old, otherwise load anything 2 years old or newer, including future dates. If you didn't need future dates then you could have just used LoadForever as the second rule .... but since LoadForever doesn't have an "include future" option you have to do it the way you wrote it. You could have used loadByPeriod (P580D+future) but the additional overlap doesn't affect the behavior since this is the lower priority rule. 2. Load anything 70 days old or newer, including future dates ... otherwise drop everything older than 90 days. Not sure what happens to the data between 70 and 90 days ... I think the rulesets always have a "LoadForever" as the last rule that you cannot delete, so I would assume everything between day 70 and 90 would also get loaded. The other two rules follow a similar pattern of interpretation. Notice my different phrasing for the BeforeBy and By options ... I am assuming that BeforeBy means prior to the period specified, and By includes the time period specified. So, for example: • BeforeBy P1D means before yesterday • By P1D means yesterday or later I am not 100% sure of this though, would welcome confirmation.

Thanks John

I believe the retention rules only determine whether or not segments get pre-loaded onto Historicals or not ... the segments still exist in deep storage unless you issue a "kill" task. Look at the bottom of the page for the link Daniel provided ... it addresses permanent deletion of data.

Hey Guys, We configured the below properties

druid.coordinator.kill.datasource.on=true
druid.coordinator.kill.maxSegments=100

as per the below reference , we have configured these properties in coordinator server -runtime. properties file /opt/druid/conf/druid/cluster/master/coordinator-overlord/runtime.properties Apart form configured properties do we need to set any additional properties in order to automate the kill operation? https://support.imply.io/hc/en-us/articles/360035308573-Deep-storage-data-segment-deletion-using-coordinator-kill But the document suggesting Once the above properties were placed , need to restart the master nodes , I am not sure about this point What are the master nodes we need to restart? or just restart the master services are sufficient. But in the below reference no where specified need to restart the master nodes. https://imply.io/blog/apache-druid-recovering-dropped-segments/

To pick up a properties change on Coordinator or Overlord, I think you have to restart all master nodes, one at a time to allow proper failover if you happen to restart the leader node.

I have restarted all the master nodes. but datasource has been wiped out.

What are the retention policies on this datasource? It is possible you have set the retention to "drop" all time chunks ... in which case the segments are still there, just not loaded for querying.