This message was deleted.
# general
s
This message was deleted.
s
You can see/update the retention rules in the Druid Console in the DataSource view in the Retention column:
The APIs call will result in changes in that view. The coordinator runs every minute by default and applies the current retention rules for each datasource, instructing the historicals to load or unload segments. Some coordinator level dynamic configs also control aspects of how this operates and are documented here.
c
Hey Sergiu! Thanks for your reply! so as far as I got from you, when we apply these retention rules, it has nothing to do with coordinator configs or change nothing there but instead it is gonna effect on data sources rather than setting them like coordinator config, right?
So in coordinator config, we can set how it should operate as for frequency and so on for retention policies we have set?
We do not want to set the retention rules for each data sources manually in UI but rather create a Python file which makes API requests to set those retention for each data sources and run the Python file via Jenkins pipeline. Do you think does this make sense? It is just because we want to have 1 common source where we can just change retention policies in case we need to change in the future. And an Airflow DAG runs periodically to apply the kill tasks. Are these common ways to achieve in our use cases?
s
@Caner Kürtür Are you planning on changing retention rules frequently?
Also, you don't need "manually" trigger kill tasks. This can also be configured to just run periodically within Druid, the basic coordinator config parameters that control this are:
Copy code
druid.coordinator.kill.on = true #turn on periodic kill tasks
druid.coordinator.kill.period    # how often to run kill tasks defaults to P1D
You can definitely use the Retention Rules API to set the rules from Python. I don't think this is common since retention rules are rarely changed but the APIs are there to be used so, if that works for you, I don't see any reason not to do it that way.
c
Hey @Sergio Ferragut! thank you so much really for the replies once again 🙂 Well actually we dont plan changes to be made frequently but just in case if we want to do it , we wanted to have a automated way having a common file for all data sources. I dont know if this is really usual way to do it
s
Maybe I misunderstood, but if it is meant to be a common retention rule for all datasources, keep in mind that the
_default
rule applies to any data sources that do not have rules of their own. So this might be an easier approach.
c
Hi @Sergio Ferragut sorry for my late reply. Thank you for your help! 🙂