This message was deleted.

Can you share a screenshot of the retention rules for that datasource? What are the default retention rules?

default - apart from the intervals mentioned for hot, everything gets loaded to cold with 2 replicas

any reason to not use loadByPeriod rules for all of them. Seems like this way there are potential holes in the timeline as you change the intervals. Like right now, you have a gap between 2022-01-21 and 2022-10-19, which means they will fall to the loadForever rule.

We are looking to load specific intervals for YoY comparisons

it does seem to be overreplicating and there are improvements on that front in 25.0,

yes the gap in intervals are intentional

Can I ask what the reason is behind the interval gap? the way it is now, segments in the gap period will be loaded into cold, then some time later, they will be loaded back into hot, and then again into cold. The segment you described as being replicated 10 times, has recently moved into the third rule timeframe, over-replication segment removal is low priority in prior releases (prior to 25.0) when compared to creating replicas in the "new rule" that applies. I'm not sure about the specific behavior on 0.18.1. I need to research changes to this since then.

The reason is to compare YoY data, we store corresponding segments on hot the gap apart from interval is always loaded to cold , they will never come back to hot again

It does seem like there is some unexpected behavior here, just to characterize this better, do you change the interval rules on a daily basis?

yes its updated via batch job

I'll do a bit of research to figure out if this has been dealt with in newer releases. Do you have

replicateAfterLoadTimeout

set to true? If you have slow historicals that are timing out their load instruction, this could lead to over-replication given that the timed out historical may will still process the load instruction.

We have never set

replicateAfterLoadTimeout

by default it is false

okay, we'll discard that as the reason. 😄

@Shilpa Sivanesan you can use the zkCli and delete the znodes /druid/segments and /druid/servedSegments

@Vijay Narayanan would it delete all the segments ? It could impact real time events

It won’t actually delete segments. It will simply allow the coordinator to recreate the information in the zookeeper. So if there are existing multiple replicas that won’t be deleted but going forward you should not face this issue.

I have tried restarting zookeeper services yesterday

restarting zk won’t delete znodes…so won’t address the issue. You can use the zkcli and examine the znodes and see if the over replication is due to zk. If it is then the best approach is to delete the znodes

sounds good , but the zk nodes were not created using zkcli

zkcli can connect to any running zk

It is the bug with retention rules, upon updating the intervals they done trigger the rules

@Shilpa Sivanesan I continue to look into this, there have been PRs that address over-replication, I'm just not certain that it addresses your case. Seems like what is happening, is that segment move from rule#1, then to rule#2, then to rule#4 and then back again to rule#3. so I think it is the case where segments are moving back one rule that is causing the over-replication. I continue to research why this might be.

Thanks @Sergio Ferragut Yes rule #3 is not executed properly Also dropforever the default rule, gets low priority and the never gets executed, which is also resulting in over replication There was a suggestion on similar issue in this channel to increase replicantLifeTime Have made the changes , will verify and update on this thread

Very interesting... thanks for sharing. It will be interesting to see how the new over-replication logic in 25.0 addresses this. It is more responsive to over-replication when segments move between tiers based on retention rules. Total target replication of the segment across tiers is evaluated, if it is higher than needed, segments are dropped from the old tier. If total replication is at target level or lower, another copy of the segment is loaded into the target tier. This logic continues until target replication is complete. In principle, when moving from rule #4 to rule #3, in 25.0 you should see: • segment is replicated twice in cold • 2 in cold (current replication level) > 1 in hot (target replication level) • 1 segment dropped from cold • 1 left in cold • target tier is still low, so 1 segment is loaded into hot • 2 (1 in cold, 1 in hot) > 1 total target reps • 1 segment is dropped from cold • 1 in hot = target tier replicas, done!

Thats sounds great , will try upgrading the version