@Mayank @Xiang Fu @Jackie pinot-server ram usage is getting increased over time without adding garbage collection params in jvmopts in pinot/values.yaml helm .. before we were using jvmopts like "*jvmOpts: "-Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/opt/pinot/gc-pinot-controller.log"*" but after migrating to jdk 11 with these jvmopts .. the pods started crashing and we have to remove these jvmopts and only using the below jvmopts ie "*jvmOpts: "-Xms2M -Xmx8G  -Xloggc/opt/pinot/gc pinot controller.logjavaagent/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml" - server* " but without using garbage collection params .. we are seeing an increase in pinot-server ram usage over time and ram is getting exhausted every day by more than half gb .. what should be the jvmopts we should provide in helm for jdk11 so that pods dont crash and gc happens properly and heap is free .. also for 16 gb server ram .. what should be xmx value for server pod..

you can still use

-XX:+UseG1GC

what should be the pinot server xmx value in jvmopts in helm if ram provided is 16 gb per pinot-server pod

@Sadim Nadeem my guess is you have a high cardinality inverted index (do you?) and that means you have a lot of `SoftReference`s, which would be cleared more aggressively by G1 on JDK8 than JDK11. If that's it, the issue should be fixed here: https://github.com/apache/pinot/pull/7493

yes I do have inverted index on many columns some of which have high cardinality

Would you be able to confirm the suspected cause with a live heap dump (

jmap -dump:live,file=dump.bin <pid>

)? If my guess is correct, there should be a lot of

ImmutableRoaringBitmap

by retained size.

ok let me check

why you say server ram is exhausted? the cmd you ran is for k8s pod memory usage, which is expected, as you configure 16gb ram, pinot uses all of them(heap and off-heap). You need to observe the jvm usage metrics to see gc behavior

Ok @Xiang Fu .. Which tool is recommended for observing jvm usage metrics

I did the last helm upgrade for pinot on friday2 PM UTC 1st october .. should I expect the fixes https://github.com/apache/pinot/pull/7493 getting pulled automatically since we have pull policy : always in the pinot helm chart pinot/values.yaml

pinot image is published daily, it’s irrelevant to the helm

how to get the latest image deployed in k8s

please read k8s basics 😛 In short, set ImagePullPolicy to Always then delete all the pods.

got it

as suggested by @Richard Startin  .. inverted indexing may cause memory leak cc: @Subbu Subramaniam

is it necessary to install MAT or JVisualVM heapdump viewer and need to confirm the same (edited)

We don't know there is a memory leak, we just know you've found a behaviour change in JDK11, and needed to increase

-Xmx

to get things working. This is why I suggested taking a heap dump - you don't need to install anything to do that, just run

jmap -dump:live,file=dump.bin <pid>

where

<pid>

is the PID of the server process. Once you have that heap dump exported, you can analyse it with JVisualVM or Eclipse MAT, and you only need to install these on your laptop so you can look at the heap dump. Knowing the top 10 types by retained size would support or conclusively disprove the suggestion that this is caused by soft reference processing changes in G1 between JDK8 and JDK11.

sure will check and update

do server/broker/controller order

thanks @Xiang Fu

yes

ok