Hello, What are the basic steps to troubleshoot a cluster. My cluster status sometimes shows Bad in UI and recovers quickly. However search and ingestion are working. No issues with CPU or Memory . Also all logs looks ok ,other than few errors due to bad query searches. So how to check whether everything is all-right?

Real-time segments will sometimes bounce around in Bad status in the UI, especially if you have multiple replicas.

Thanks . You are correct I do have multiple replicas and using realtime. Great to know that this is a normal behavior and good that a PR is already there.

That’s right , it’s a minor bug in the UI logic

@Kishore G I can also see the memory usage is growing after my last restart. I expect Pinot to trigger a GC? or is it MMAP . This is my Graph

It’s mmap

How are you getting memory usage?

from kubernetes metrics

That’s ok - OS will manage pod memory

ohh ok, makes sense. but the memory increase is a normal behavior right? and it will purged by OS at some point?

Yes

Great Thanks 👍 , will monitor and see how it goes.

Yes, OS is good at managing it...