Hi Team, We had an incident where heap memory of one of our pinot servers was above 90%. This caused downtime in production. How can we prevent it in the future? We are using pinot 1.4.0 currently on k8s and value.yaml is in the comments.

Do you know why it was high?

We are still investigating. Maybe we are running a lot of multistage engine with joins and looks like our garbage collection config is not optimal. We will also migrate to 1.5 and will test this https://docs.pinot.apache.org/operate-pinot/tuning/oom-protection-using-automatic-query-killing?q=PerQueryCPUMemAccountantFactory I am trying to understand more how garbage collection works and how we can optimize JVM_OPTS for our setup

I'd say get a baseline with 1.5 before modifying GC

So we rolled out 1.5 but the issue still exists. I have analyzed some prometheus metrics See the metric {pod=~"$pod", name="jvm_memory_pool_collection_used_bytes"} We can clearly see a spike in GI Old Gen jvm heap memory in pinot-server-0. It went from 8GB baseline to 48GB in 2 days two times. We are not able to understand why garbage collector is not cleaning it up. As a band aid, we were restarting the cluster every Sunday night so that all heap memory gets cleared out. But now, even that is not enough 😕