Hi folks .. most probably because of having inverted indexing on high cardinality data .. some memory leak is happening on pinot server pods .. and ram usage gradually increases by few hundred mb every day .. temporary resolution to free heap/ram memory we are thinking of is trying to restart pinot server pods on weekly basis using command:- kubectl delete po dev-pinot-server-0 dev-pinot-server-1 dev-pinot-server-2 -n dev-pinot will it cause any data loss or table and schema and data in table will remain intact after restart/bouncing of pinot server pods.. *ref:*https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/ Delete Pods You can perform a graceful pod deletion with the following command:

kubectl delete pods <pod>

Can you create a Github issue with as much details as possible for helping debug and fix the memory issue you are reporting, that would be great. Note, that Pinot is running in huge deployments in production and this issue hasn't been reported, so it might be specific to your usecase or setup. So if you can provide all the details, we can help debug/fix the issue.

@Mayank we need to take heap dump/thread dump in case we need to create a github issue ? also can somebody confirm if we perform kubectl delete po  dev-pinot-server-0  dev-pinot-server-1  dev-pinot-server-2  -n   dev-pinot will it cause any data loss or table and schema and data in table will remain intact after restart/bouncing of pinot server pods..

Yes, that would be good, you can also provide info like:

1. Graphs
2. Release you are using
3. Use case related info (read/write qps)
4. Is leak in offline or realtime, if RT, the server configs, etc

we are using realtime

How do you get ram usage?

By using kubectl top po -n dev-pinot

it’s k8s container pod memory usage. You should track jvm memory usage and gc happening.

on grafana .aftrer 10 or 15 minutes.. the pinot jvm memory chart stops showing any metrics while controller broker ram usage is always available on the time series chart for all the time .. thus unable to track by grafana using prometheus .. any other recommended way like jconsole or something ..

whats your k8s pod container size?

Pinot server pod k8s pods ram and cpu configs :- resources: requests: memory: "16G" cpu: "5" limits: memory: "16G" cpu: "5"

yes, and the kubectl top gives you the container memory usage, it will leverage offheap memory as well, so you will see more memory usage for sure. As long as you don’t encounter pod oom kill then you are fine.