This message was deleted.

When this has happened to me, it was an accidentally created subscription on a high volume topic. It sounds like you've tried to auto-delete these, but that could still be contributing. Getting the broker level stats can often provide a lot of helpful information around backlogs.

This problem could be because my bookie direct memory is always at high end, Initially DirectMemory with bookies was set to 4GB for a long time, 3 months back we saw that DirectMemory is exhausted so we increased it and ended up assigning 20GB to it to make the cluster stable. We see that 20GB is also completely utilised and our bookie kept getting restarted. I believe Direct Memory could be one reason of our storage not getting cleaned up.

That seems like a lot of direct memory. How large are your ledgers? Is compaction correlated with high memory utilization?

I am using default config for ledgers as well as read cache & do not have any requirement of keeping data cache ready. But I do see out of 20GB, read cache is using 5GB. More than what is assigned by default which is 25% of default memory. My write cache peak is also < 15MB. Not sure where rest of the memory is going.