https://pinot.apache.org/ logo
Join Slack
Powered by
Hello, What are the basic steps to troubleshoot a ...
# troubleshooting
m
Hello, What are the basic steps to troubleshoot a cluster. My cluster status sometimes shows Bad in UI and recovers quickly. However search and ingestion are working. No issues with CPU or Memory . Also all logs looks ok ,other than few errors due to bad query searches. So how to check whether everything is all-right?
w
Real-time segments will sometimes bounce around in Bad status in the UI, especially if you have multiple replicas.
That's been my experience, anyway
I found this open issue, so I assumed it wasn't worth worrying about: https://github.com/apache/incubator-pinot/issues/6137
m
Thanks . You are correct I do have multiple replicas and using realtime. Great to know that this is a normal behavior and good that a PR is already there.
k
That’s right , it’s a minor bug in the UI logic
m
@Kishore G I can also see the memory usage is growing after my last restart. I expect Pinot to trigger a GC? or is it MMAP . This is my Graph
image.png
k
It’s mmap
➕ 1
What your GC setting
m
Copy code
jvmOpts: "-Xms512M -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/dev/stdout -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 "
Pod mem is 26G
k
How are you getting memory usage?
That graph looks like system usage not jvm
m
from kubernetes metrics
Thats correct that Pod is only running single pinot server
k
That’s ok - OS will manage pod memory
Jvm will be under 4 gb
m
ohh ok, makes sense. but the memory increase is a normal behavior right? and it will purged by OS at some point?
k
Yes
m
Great Thanks 👍 , will monitor and see how it goes.
k
Yes, OS is good at managing it...
👍 1
Open in Slack
PreviousNext
Powered by