Hi Team, SErver JVM was using more than allocated...
# troubleshooting
a
Hi Team, SErver JVM was using more than allocated memory (8GB), which was leading to server downtime and recovery when querying we increased JVM of server from 8GB to 10GB and tried restarting servers. We faced OOMKilled in zookeeper and crashloop backoff after the server restart. we upgraded zookeeper memory from 8GB to 16GB even then the OOMKilled is happening in Zookeeper and still going into crashloop backoff Any idea how to recover Zookeeper from this state? We keep trying server and zookeeper restarts but not helping Zookeeper log
Copy code
zookeeper 18:08:23.22 INFO  ==>
zookeeper 18:08:23.22 INFO  ==> ** Starting ZooKeeper setup **
zookeeper 18:08:23.31 WARN  ==> You have set the environment variable ALLOW_ANONYMOUS_LOGIN=yes. For safety reasons, do not use this flag in a production environment.
zookeeper 18:08:23.32 INFO  ==> Initializing ZooKeeper...
zookeeper 18:08:23.32 INFO  ==> User injected custom configuration detected!
zookeeper 18:08:23.32 INFO  ==> Deploying ZooKeeper with persisted data...
zookeeper 18:08:23.33 INFO  ==> ** ZooKeeper setup finished! **
 
zookeeper 18:08:23.34 INFO  ==> ** Starting ZooKeeper **
/opt/bitnami/java/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg
Removing file: Apr 7, 2025, 2:40:30 PM  /bitnami/zookeeper/data/version-2/log.60416
Removing file: Apr 7, 2025, 2:45:42 PM  /bitnami/zookeeper/data/version-2/snapshot.604c4
2025-04-07 18:08:28,533 [myid:1] - ERROR [main:o.a.z.s.p.Util@166] - Last transaction was partial.
saurabhpa@ip-10-254-1-68:~/Pinot-prod/pinot/helm/pinot$
Server log (That we tried to restart)
Copy code
2025/04/07 18:06:05.437 INFO [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] SASL config status: Will not attempt to authenticate using SASL (unknown error)
2025/04/07 18:06:05.437 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] Session 0x100013f2a470008 for server pinot-zookeeper/172.20.15.241:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
2025/04/07 18:06:06.607 INFO [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] Opening socket connection to server pinot-zookeeper/172.20.15.241:2181.
2025/04/07 18:06:06.607 INFO [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] SASL config status: Will not attempt to authenticate using SASL (unknown error)
2025/04/07 18:06:06.607 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] Session 0x100013f2a470008 for server pinot-zookeeper/172.20.15.241:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
CC : @Saurabh Patil
x
in k8s, don’t set memory limit, only keeps the request
s
Hi @Xiang Fu please check the snip.
x
this is good
what’s your jvm opts
s
also we attached server and zookeeper logs in original message
## Size (in MB) for the Java Heap options (Xmx and Xms) ## This env var is ignored if Xmx an Xms are configured via
zookeeper.jvmFlags
heapSize: "1024"
@Xiang Fu For Zookeeper we are not using the jvmOpts.
x
then you should give more heap for zookeeper as you reserved 14g ram
s
can you recommend heap
x
try 8g(8192) for now
s
Sure!
x
double it when you see oom 🙂
make sure you put it both places k8s resources and zk config
s
@Xiang Fu Still its not working. Throwing OOMKilled for zookeeper with 8GB heap.
after restarting pinot-server-pod.
@Xiang Fu getting OOMKilled error for Zookeeper, could you suggest solution for this.
p
@Saurabh Patil Check the info logs for servers by exec into the server pod. If your server was killed, on restart, it tries to first kill the old session on zookeeper and update the status of esch segment. If you have too many segments, it will overload the zookeeper pods on server start At least this was the case on our side. If high segment count is the issue on your side as well, consider deleting a few segments and the cluster should heal. After this issue is fixed, you need to think about merging segments to reduce the overall segment count
x
can you check the zookeeper statefulset resource section
so put request 16g memory if you use 8g heap
s
It worked. @Xiang Fu Thank you!