Hello Team,
We are encountering an issue where our...
# troubleshootingb
Badhusha Muhammed
10/30/2025, 4:17 PMHello Team,
We are encountering an issue where our Pinot servers are timing out when attempting to establish a session with Zookeeper. This is causing the Pinot Servers to crash (or go down).
Although the server attempts to iteratively establish a new connection, the process continues to time out until we manually restart the server instance.
A similar scenario can be found in the following GitHub issue: https://github.com/apache/pinot/issues/4686.
1. The initial issue between the Pinot Server and Zookeeper was session expiration.
2. Regardless of the underlying issue (e.g., Zookeeper latency, GC pauses blocking the main thread), Pinot should be capable of automatically re-establishing the connection once the problem is resolved. Instead, we are forced to manually restart the server to restore a healthy Zookeeper session.
As a result , the server is being removed from the LIVE_INSTANCE metadata and registered as DEAD.
Badhusha Muhammed
10/30/2025, 4:19 PMWe have a total 5 REALTIME tables , with around 360K segments
Badhusha Muhammed
10/30/2025, 4:22 PMā¢ What can cause this , Continues timeout from Pinot to Zookeeper
ā¢ Why pinot servers are not able to re establish a healthy connection until we restart
Badhusha Muhammed
10/30/2025, 4:23 PMFrom Pinot Server
Copy code
WARN [Start a Pinot [SERVER]-SendThread(zookeeper_host:2181)] Client session timed out, have not heard from server in 33859ms for session id 0x206e70f8070009e Copy code
INFO [NIOWorkerThread-13:ZooKeeperServer@828] - Invalid session 0x206e70f8070009e for client /pint_server:30252, probably expiredBadhusha Muhammed
10/31/2025, 7:30 AM@Mayank, @Xiang Fu Can you please help here ..?
x
Xiang Fu
10/31/2025, 7:32 AM
see if it's due to znode side issue
Xiang Fu
10/31/2025, 7:32 AM
b
Badhusha Muhammed
10/31/2025, 7:40 AMWe have increase the znode size of zookeeper, The znode is currently at 10MB , and there are other service actively using this zookeeper, Where we are not seeing this timeout , and not all the Pinot Servers are affected due to this
One other major concern is , why Server is not able to re establish the connection, Until we restart them. Similar to this Github issue ->> https://github.com/apache/pinot/issues/4686. This leads to a manual intervention to make the server healthy again .
x
Xiang Fu
10/31/2025, 8:51 AM
did you observe any log lines like:
java.io.ioexception: packet len 1250829 is out of range!Xiang Fu
10/31/2025, 8:51 AM
usually this will break the client connection immediately
b
Badhusha Muhammed
10/31/2025, 8:59 AMNo, Not observed any such issue , Packets are with in jut.maxbuffer threshold
Badhusha Muhammed
10/31/2025, 9:02 AMIn fact, We observed this issue in the past , When we were running with the default (1MB, At that time packet length pinot trying to process is around 3MB), So we increase this to 10MB, (Both in zookeeper and Pinot)
Badhusha Muhammed
10/31/2025, 9:05 AMWe see this in the log
Copy code
WARN [ZkClient-EventThread-114-zookeeper_host:2181] StateModel - Default reset method invoked. Either because the process longer own this resource or session timedoutBadhusha Muhammed
10/31/2025, 9:19 AMThis is keep on coming in log,
Copy code
2025-10-30 07:44:30.726 INFO [Start a Pinot [SERVER]-SendThread(zookeeper-host:2181)] ClientCnxn - Opening socket connection to server zookeeper-host/zookeeper-ip:2181.
2025-10-30 07:44:57.993 INFO [Start a Pinot [SERVER]-SendThread(zookeeper-host:2181)] ClientCnxn - Socket connection established, initiating session, client: /zookeeper-ip:59202, server: zookeeper-host/zookeeper-ip:2181
2025-10-30 07:45:15.623 WARN [Start a Pinot [SERVER]-SendThread(zookeeper-host:2181)] ClientCnxn - Session 0x104717649aa0090 for server zookeeper-host/zookeeper-ip:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
2025-10-30 07:54:45.770 INFO [Start a Pinot [SERVER]-SendThread(zookeeper-host:2181)] ClientCnxn - Opening socket connection to server zookeeper-host/zookeeper-ip:2181.
2025-10-30 07:54:45.770 INFO [Start a Pinot [SERVER]-SendThread(zookeeper-host:2181)] ClientCnxn - Socket connection established, initiating session, client: /zookeeper-ip:26280, server: zookeeper-host/zookeeper-ip:2181
2025-10-30 07:54:45.771 WARN [Start a Pinot [SERVER]-SendThread(zookeeper-host:2181)] ClientCnxn - Session 0x104717649aa0093 for server zookeeper-host/zookeeper-ip:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.x
Xiang Fu
10/31/2025, 11:30 AM
hmm, which pinot and zookeeper version are you running
Xiang Fu
10/31/2025, 11:30 AM
and did you observe any full gc happening ?
Xiang Fu
10/31/2025, 11:32 AM
also what's the zookeeper tick/init/sync settings ?
b
Badhusha Muhammed
10/31/2025, 12:09 PMYes, we did faced a full gc at Pinot server side , But the issue is that
Pinot GC Started at 07hr:01 - 08hr:48
But the , Zookeeper connection failure started happening atn 07hr:44 (Which you can see it from log), Which is 43 min after Full GC started
Copy code
tickTime=3000
tickTime=3000
syncLimit=5j
Johan Adami
10/31/2025, 1:25 PMfrom the last time timeouts came up, https://apache-pinot.slack.com/archives/C011C9JHN7R/p1757857117375799?thread_ts=1757840808.452529&cid=C011C9JHN7R
š 1
b
Badhusha Muhammed
11/04/2025, 8:05 AMHi @Xiang Fu, So if the Pinot Servers are not able to automatically create the session (Due to some reason), and the solution is only to restart Pinot servers, Is it possible to have an automatic stop or restart of the server, Maybe after a couple of reties , Or is this addressed in any of the Pinot Versions ,
Stack Versions in our Env
Zookeeper : 3.5.10
Pinot : 1.2
x
Xiang Fu
11/04/2025, 8:09 AM
hmmm, have you tried to upgrade zookeeper to 3.7+?
b
Badhusha Muhammed
11/07/2025, 5:50 PMNo , We haven't tried upgrading the zookeeper.
Badhusha Muhammed
11/07/2025, 5:54 PMBut is there any specific feature we are looking at , in Zookeeper 3.7+, which we are missing in 3.5..?