https://pinot.apache.org/ logo
m

Matt

01/20/2021, 2:52 PM
Hello, What are the basic steps to troubleshoot a cluster. My cluster status sometimes shows Bad in UI and recovers quickly. However search and ingestion are working. No issues with CPU or Memory . Also all logs looks ok ,other than few errors due to bad query searches. So how to check whether everything is all-right?
w

Will Briggs

01/20/2021, 3:23 PM
Real-time segments will sometimes bounce around in Bad status in the UI, especially if you have multiple replicas.
That's been my experience, anyway
I found this open issue, so I assumed it wasn't worth worrying about: https://github.com/apache/incubator-pinot/issues/6137
m

Matt

01/20/2021, 3:35 PM
Thanks . You are correct I do have multiple replicas and using realtime. Great to know that this is a normal behavior and good that a PR is already there.
k

Kishore G

01/20/2021, 4:09 PM
That’s right , it’s a minor bug in the UI logic
m

Matt

01/20/2021, 4:18 PM
@Kishore G I can also see the memory usage is growing after my last restart. I expect Pinot to trigger a GC? or is it MMAP . This is my Graph
message has been deleted
k

Kishore G

01/20/2021, 4:19 PM
It’s mmap
1
What your GC setting
m

Matt

01/20/2021, 4:22 PM
Copy code
jvmOpts: "-Xms512M -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/dev/stdout -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 "
Pod mem is 26G
k

Kishore G

01/20/2021, 4:23 PM
How are you getting memory usage?
That graph looks like system usage not jvm
m

Matt

01/20/2021, 4:23 PM
from kubernetes metrics
Thats correct that Pod is only running single pinot server
k

Kishore G

01/20/2021, 4:24 PM
That’s ok - OS will manage pod memory
Jvm will be under 4 gb
m

Matt

01/20/2021, 4:26 PM
ohh ok, makes sense. but the memory increase is a normal behavior right? and it will purged by OS at some point?
k

Kishore G

01/20/2021, 4:26 PM
Yes
m

Matt

01/20/2021, 4:27 PM
Great Thanks 👍 , will monitor and see how it goes.
k

Kishore G

01/20/2021, 4:27 PM
Yes, OS is good at managing it...
👍 1