Hello, We have a strange behavior on our coordinator nodes (

v31.0.2

) Everything is really stable but after some time, coordinator nodes monitoring complains about memory usage. Note that we never have any OOM kill (maybe we don't wait enough time for that 😄 ) After some researchs, we found out that the RSS was increasing more and more over time. (on the attached graph, rss is the blue line. It starts increasing after the node became the leader) A simple restart of the node fixes the alerts (but not the root cause) It seems that some resources are never getting freed. We also found out about this article : blog.malt.engineering/java-in-k8s-how-weve-reduced-memory-usage-without-changing-any-code… which is about the glibc malloc implementation that causes fragmentation and resources never properly cleaned. Potential helper would be to set

MALLOC_ARENA_MAX

to a lower value than its default (or switch to

jemalloc

) Is my troubleshooting correct or did I miss something ? Has this issue already been encountered by someone ?

I'm also having issue with Broker memory growing which is likely attributable to glibc malloc arena bloat. Did you end up making any changes? Did it work?

I'm currently testing jemalloc and it causes no issue. But the behavior we encounter is not easily repeatable in non prod (since we have far less activity). I'm planning to deploy it in production by the end of this week or next week