We are sufferring from timeout issues with ZK sessions lead Apache Pinot #troubleshooting

We are sufferring from timeout issues with ZK sess...

Luis P Fernandes

09/14/2025, 9:06 AM

We are sufferring from timeout issues with ZK sessions leading to issues constant segments marked as UPDATING. We suffered and IP issue with one of our servers, and once that was resolved it triggered a large number of segments reload, ater that we started to experience timeout issues with ZK sessions: Pinot Server Logs: 2025/09/13 193221.866 INFO [ZkClient] [Start a Pinot [SERVER]-EventThread] zkclient 3, zookeeper state changed ( Expired ) 2025/09/13 193221.866 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(100.100.80.80:2191)] Session 0x10008afb572019b for server aiops-5g-pinot-test-zookeeper1/100.100.80.80:2191, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException. org.apache.zookeeper.ClientCnxn$SessionExpiredException: Unable to reconnect to ZooKeeper service, session 0x10008afb572019b has expired at org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1439) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocket.readConnectResult(ClientCnxnSocket.java:150) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:86) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1291) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] 2025/09/13 193221.867 INFO [ZooKeeper] [Start a Pinot [SERVER]-EventThread] Initiating client connection, connectString=100.100.80.80:2191 sessionTimeout=30000 watcher=org.apache.helix.zookeeper.impl.client.ZkClient@529eba37 2025/09/13 193221.867 INFO [ClientCnxnSocket] [Start a Pinot [SERVER]-EventThread] jute.maxbuffer value is 40000000 Bytes ZK Logs: Sep 14 122633 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122633,971 [myid:] - INFO [Snapshot Thread:o.a.z.s.ZooKeeperServer@589] - Snapshot taken in 1874 ms Sep 14 122659 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122659,329 [myid:] - INFO [SessionTracker:o.a.z.s.ZooKeeperServer@730] - Expiring session 0x100002eb7b20024, timeout of 30000ms exceeded Sep 14 122701 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122701,329 [myid:] - INFO [SessionTracker:o.a.z.s.ZooKeeperServer@730] - Expiring session 0x100002eb7b20036, timeout of 30000ms exceeded Sep 14 122704 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122704,553 [myid:] - INFO [NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1177] - Invalid session 0x100002eb7b20024 for client /100.100.25.106:45014, probably expired Sep 14 122705 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122705,332 [myid:] - INFO [NIOWorkerThread-10:o.a.z.s.NIOServerCnxn@342] - Unable to read additional data from client, it probably closed the socket: address = /100.100.100.131:38872, session = 0x100002eb7b20039 Sep 14 122723 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122723,330 [myid:] - INFO [SessionTracker:o.a.z.s.ZooKeeperServer@730] - Expiring session 0x100002eb7b20039, timeout of 30000ms exceeded Sep 14 122728 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122728,859 [myid:] - INFO [NIOWorkerThread-1:o.a.z.s.NIOServerCnxn@342] - Unable to read additional data from client, it probably closed the socket: address = /100.100.5.153:39416, session = 0x100002eb7b20031 Sep 14 122731 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122731,516 [myid:] - INFO [NIOWorkerThread-7:o.a.z.s.NIOServerCnxn@342] - Unable to read additional data from client, it probably closed the socket: address = /100.100.100.131:41740, session = 0x0 Sep 14 122753 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122753,329 [myid:] - INFO [SessionTracker:o.a.z.s.ZooKeeperServer@730] - Expiring session 0x100002eb7b20031, timeout of 30000ms exceeded Sep 14 122814 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122814,494 [myid:] - INFO [NIOWorkerThread-3:o.a.z.s.NIOServerCnxn@342] - Unable to read additional data from client, it probably closed the socket: address = /100.100.105.251:34936, session = 0x0 Sep 14 122820 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122820,024 [myid:] - INFO [SyncThread0o.a.z.s.p.FileTxnLog@291] - Creating new log file: log.c9d9be7 Sep 14 122820 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122820,025 [myid:] - INFO [Snapshot Thread:o.a.z.s.p.FileTxnSnapLog@480] - Snapshotting: 0xc9d9b19 to /srv/zookeeper/data/version-2/snapshot.c9d9b19 Sep 14 122821 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122821,992 [myid:] - INFO [Snapshot Thread:o.a.z.s.ZooKeeperServer@589] - Snapshot taken in 1968 ms Sep 14 122838 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122838,721 [myid:] - INFO [Snapshot Thread:o.a.z.s.p.FileTxnSnapLog@480] - Snapshotting: 0xc9e9a9b to /srv/zookeeper/data/version-2/snapshot.c9e9a9b Sep 14 122838 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122838,722 [myid:] - INFO [SyncThread0o.a.z.s.p.FileTxnLog@291] - Creating new log file: log.c9e9a9d Sep 14 122840 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122840,437 [myid:] - INFO [Snapshot Thread:o.a.z.s.ZooKeeperServer@589] - Snapshot taken in 1716 ms Sep 14 122910 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122910,961 [myid:] - INFO [NIOWorkerThread-5:o.a.z.s.NIOServerCnxn@342] - Unable to read additional data from client, it probably closed the socket: address = /100.100.123.106:54930, session = 0x100002eb7b20026 Sep 14 122916 aiops-5g-pinot-test-zookeeper1 zkServer.sh[8056]: 2025-09-14 122916,001 [myid:] - INFO [NIOWorkerThread-6:o.a.z.s.NIOServerCnxn@342] - Unable to read additional data from client, it probably closed the socket: address = /100.100.5.153:37504, session = 0x0 We have increase the Djute.maxbuffer to 4MB but we still cannot bring the cluste back to operation. Is there an option to increase the sessions timeout as all the flags: -zookeeper.connection.timeout, zk.session.timeout.ms, helix.zookeeper.session.timeout dont seem to work. Any suggestions would be welcomed, thanks.

Johan Adami

09/14/2025, 1:36 PM

in the past, we’ve only had to increase session timeouts because of GC running > 30s tripping the default timeout. we don’t do it anymore since moving to java 21 + gZGC

Johan Adami

09/14/2025, 1:38 PM

but the way we increased timeout was: •

-Dhelixmanager.waitForConnectedTimeout=180000 -Dzk.session.timeout=60000

•

<http://zk.client.session.timeout.ms|zk.client.session.timeout.ms>=60000

in the pinot

.conf

file You have to set both (conf file and java command). i think the second one is what you’re missing that still leads to disconnects from servers contacting ZK directly rather than through helix

🙌 1

Mayank

09/14/2025, 4:28 PM

How many segments do you have? What’s the IP issue you are referring to? You should not use ip as instance name

Luis P Fernandes

09/14/2025, 7:08 PM

The IP/Networking issue was one of our server VMs lost connectivity due to networking issue for a few hours. As part of the diagnostic of this issue we found that we have way too many small segments and we will make changes and do rollups to reduce that. However at the moment we cant get the cluster up and running due to this timeout issues, I have managed to raised the timeout to 90s but still keep haveing drop outs and cant get the segments in a good state. I see a lot of the following errors that precede the timeouts: Sep 14 205404 aiops-5g-pinot-test-server6 pinot-admin.sh[162620]: 2025/09/14 205404.053 ERROR [HelixTask] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Exception while executing a message. org.apache.helix.zookeeper.zkclient.exception.ZkInterruptedException: java.lang.InterruptedException msgId: f60e1e7c-8675-4e76-9278-d3165ec608aa type: STATE_TRANSITION Sep 14 205404 aiops-5g-pinot-test-server6 pinot-admin.sh[162620]: org.apache.helix.zookeeper.zkclient.exception.ZkInterruptedException: java.lang.InterruptedException

Luis P Fernandes

09/15/2025, 11:26 AM

We have 6 servers, and we are currently doing a strategy to bring one server up, one at a time in sequence to reduce the pressure and the number of paralel segments being reload at the same time. Anyone has any suggestions ?

Mayank

09/15/2025, 1:49 PM

We need to first understand why this is happening, and where the choke point is. Without that you might just be trying out random things. Is the disconnect because server GCs or ZK overloaded, or large ZNodes, or some network issues? How many segments is too many for you?

Xiang Fu

09/15/2025, 2:46 PM

You may also need to increase the jutebuffer size everywhere.

Xiang Fu

09/15/2025, 2:47 PM

https://docs.pinot.apache.org/reference/troubleshooting/troubleshoot-zookeeper See if you saw any error like:

Copy code

java.io.ioexception: packet len 1250829 is out of range!

Luis P Fernandes

09/16/2025, 3:34 AM

Hi guys, so first regarding the number of large segments, we have very small tables only a few hundred MBs that have more than 300 segments so clearly this is not optimal, as for total that is currently about 190k for the cluster on a single ZK instance. We already raised the jutebuffer level everywhere since prior to this incidente we had another issue related to that. Regarding the disconnects they were seeing a lot of segmente reload errors such as: Apache pinot error on journal: Sep 14 202507 aiops-5g-pinot-test-server2 pinot-admin.sh[215712]: Sep 14, 2025 82507 PM org.glassfish.jersey.server.ServerRuntime$Responder writeResponse Sep 14 202507 aiops-5g-pinot-test-server2 pinot-admin.sh[215712]: SEVERE: An I/O error has occurred while writing a response message entity to the container output stream. Sep 14 202507 aiops-5g-pinot-test-server2 pinot-admin.sh[215712]: org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Write timeout exceeded when trying to flush the data For every of those errors the segment where it would happen would drop connecitivyt to ZK and when it started new sessions it would restart to reload all its segments. This was happening constantly accross all servers. this keep in a sort of loop condition over 3 days. We have a slow S3 backed cold storage so we think we might have hit some limits with all 6 servers trying to read the cold data at the same time hence our strategy to reload one at a time. We went ahead with the strategy to bring one server at a time and we managed to recover/reload the segments in 5 of the servers so only a single one remaining. We will add 2 more ZK nodes for resiliency and to improve performance. It would be great if there was some recommendations on session timeout values, JVM recommendations and general envioroment tune up for Pinot.

Mayank

09/16/2025, 3:47 AM

These are band aids, you need to fix the root cause:

It would be great if there was some recommendations on session timeout values, JVM recommendations and general envioroment tune up for Pinot.

Mayank

09/16/2025, 3:48 AM

I am still unclear based on the info you provided on what's the root cause here.

Luis P Fernandes

09/16/2025, 4:16 AM

We see a lot of IO related errors when the segments were all being reload at the same time, after we reduce that they have stopped. What else should I check to find the root cause, any suggestion of where to look?

Mayank

09/16/2025, 4:16 AM

What specific step is causing that IO error?

Mayank

09/16/2025, 4:17 AM

Is the server trying to build indexes?

Mayank

09/16/2025, 4:17 AM

What's your server cpu/mem and jvm configs?

Luis P Fernandes

09/16/2025, 4:34 AM

This is an example from the logs when the error occurred: Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: 2025/09/15 090057.905 ERROR [ZKMetadataProvider] [HelixTaskExecutor-message_handle_thread_10] Caught exception while getting schema: agg2_kalix_volte Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: org.apache.helix.zookeeper.zkclient.exception.ZkInterruptedException: java.lang.InterruptedException Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.zookeeper.zkclient.ZkClient.retryUntilConnected(ZkClient.java:2097) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.zookeeper.zkclient.ZkClient.readData(ZkClient.java:2244) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.zookeeper.zkclient.ZkClient.readData(ZkClient.java:2235) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.manager.zk.ZkBaseDataAccessor.get(ZkBaseDataAccessor.java:495) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.manager.zk.ZkCacheBaseDataAccessor.get(ZkCacheBaseDataAccessor.java:397) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.store.zk.AutoFallbackPropertyStore.get(AutoFallbackPropertyStore.java:101) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.pinot.common.metadata.ZKMetadataProvider.getSchema(ZKMetadataProvider.java:608) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.pinot.common.metadata.ZKMetadataProvider.getTableSchema(ZKMetadataProvider.java:669) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.pinot.core.data.manager.BaseTableDataManager.fetchTableConfigAndSchema(BaseTableDataManager.java:356) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.pinot.segment.local.data.manager.TableDataManager.fetchIndexLoadingConfig(TableDataManager.java:302) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.pinot.core.data.manager.offline.OfflineTableDataManager.doAddOnlineSegment(OfflineTableDataManager.java:50) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.pinot.core.data.manager.BaseTableDataManager.addOnlineSegment(BaseTableDataManager.java:330) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOnlineSegment(HelixInstanceDataManager.java:259) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:131) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023d a298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: Caused by: java.lang.InterruptedException Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/java.lang.Object.wait0(Native Method) ~[?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/java.lang.Object.wait(Object.java:366) ~[?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at java.base/java.lang.Object.wait(Object.java:339) ~[?:?] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1605) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1577) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1970) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2000) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.zookeeper.zkclient.ZkConnection.readData(ZkConnection.java:196) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.zookeeper.zkclient.ZkClient$12.call(ZkClient.java:2248) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.zookeeper.zkclient.ZkClient$12.call(ZkClient.java:2244) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: at org.apache.helix.zookeeper.zkclient.ZkClient.retryUntilConnected(ZkClient.java:2079) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186]: ... 23 more Sep 15 090057 aiops-5g-pinot-test-server1 pinot-admin.sh[3116186] I dont think it was building indexes but not sure. As for the server specs they are the following: EC2 *r5.2xlarge - 8 cores (*Intel Xeon Platinum 8175) | 64GB RAM | 10 Gibps Network JVM: -Xms30G -Xmx40G

Mayank

09/16/2025, 4:36 AM

you are giving too much heap, but that isn't the issue here. You should reduce it to 16 or 32 GB (max)

Luis P Fernandes

09/16/2025, 4:55 AM

Here is another example: From the logs it attempted to fetch a segment and the error started from there ./pinot-all-2025-09-14-1.log:2025/09/14 233558.820 WARN [nokia_cnf_kpireferencepoints5_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Failed to initialize SegmentDirectory for segment: nokia_cnf_kpireferencepoints5__0__39__20250906T1613Z with error: Directory '/srv/pinot_hot_data/PinotServer/index/nokia_cnf_kpireferencepoints5_REALTIME/nokia_cnf_kpireferencepoints5__0__39__20250906T1613Z' does not exist. ./pinot-all-2025-09-14-1.log:2025/09/14 233558.823 WARN [HttpSegmentFetcher] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Caught exception while downloading segment from: http://100.100.35.178:9000/segments/nokia_cnf_kpireferencepoints5/nokia_cnf_kpireferencepoints5__0__39__20250906T1613Z to: /srv/pinot_hot_data/PinotServer/index/nokia_cnf_kpireferencepoints5_REALTIME/tmp/tmp-nokia_cnf_kpireferencepoints5__0__39__20250906T1613Z-fc7a3d4a-44aa-48b1-9439-3bcdc572d874/nokia_cnf_kpireferencepoints5__0__39__20250906T1613Z.tar.compressed ./pinot-all-2025-09-14-1.log:2025/09/14 233558.824 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Exception while executing a state transition task nokia_cnf_kpireferencepoints5__0__39__20250906T1613Z ./pinot-all-2025-09-14-1.log:2025/09/14 233558.824 ERROR [StateModel] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Default rollback method invoked on error. Error Code: ERROR ./pinot-all-2025-09-14-1.log:2025/09/14 233558.888 ERROR [HelixTask] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Message execution failed. msgId: fe5967ad-7810-4630-be93-bea1b32b79ff, errorMsg: java.lang.reflect.InvocationTargetException ./pinot-all-2025-09-14-1.log:2025/09/14 233558.924 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Skip internal error. errCode: ERROR, errMsg: null Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: 2025/09/14 233558.824 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Exception while executing a state transition task nokia_cnf_kpireferencepoints5__0__39__20250906T1613Z Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: java.lang.reflect.InvocationTargetException: null Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118) ~[?:?] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 1 attempts Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.common.utils.fetcher.HttpSegmentFetcher.fetchSegmentToLocal(HttpSegmentFetcher.java:70) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:127) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:135) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:168) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:811) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:761) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndLoadSegment(BaseTableDataManager.java:405) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.core.data.manager.BaseTableDataManager.addNewOnlineSegment(BaseTableDataManager.java:376) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.doAddOnlineSegment(RealtimeTableDataManager.java:457) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.core.data.manager.BaseTableDataManager.addOnlineSegment(BaseTableDataManager.java:330) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOnlineSegment(HelixInstanceDataManager.java:259) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:131) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023d> Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: ... 9 more Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: 2025/09/14 233558.824 ERROR [StateModel] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Default rollback method invoked on error. Error Code: ERROR Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: 2025/09/14 233558.888 ERROR [HelixTask] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Message execution failed. msgId: fe5967ad-7810-4630-be93-bea1b32b79ff, errorMsg: java.lang.reflect.InvocationTargetException Sep 14 233558 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: 2025/09/14 233558.924 ERROR [HelixStateTransitionHandler] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Skip internal error. errCode: ERROR, errMsg: null Sep 14 233747 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: Sep 14, 2025 113747 PM org.glassfish.jersey.server.ServerRuntime$Responder writeResponse Sep 14 233747 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: SEVERE: An I/O error has occurred while writing a response message entity to the container output stream. Sep 14 233747 aiops-5g-pinot-test-server3 pinot-admin.sh[337454]: org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed

Mayank

09/16/2025, 5:53 AM

What’s the servers data dir? I hope it is not /tmp And there’s enough disk space?

Xiang Fu

09/16/2025, 5:55 AM

how many controllers you have and can you check one segment and see what's the download url

Xiang Fu

09/16/2025, 5:57 AM

if it's http download url then it means all the pinot segments are stored in controller local disk, which is not ideal, better to make it deep store like s3

Luis P Fernandes

09/16/2025, 6:26 AM

Each server has a dedicated storage local on the local nvme disk /srv/pinot_hot_data/ while the the cold data is an Lustre NFS with an S3 backend. We have 2 controllers with the dedicated NFS(EFS based) shared between the two controllers where the controller data is stored.

Xiang Fu

09/16/2025, 6:39 AM

i see, then for this download issue I think increase http timeout might help

Xiang Fu

09/16/2025, 6:39 AM

also did you see any logs from controller side ?

Luis P Fernandes

09/16/2025, 7:00 AM

I see errors like the following: Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.151 ERROR [CompletionServiceHelper] [async-task-thread-25381] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.171 ERROR [CompletionServiceHelper] [async-task-thread-25382] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.183 ERROR [CompletionServiceHelper] [async-task-thread-25401] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.185 ERROR [CompletionServiceHelper] [async-task-thread-25414] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.188 ERROR [CompletionServiceHelper] [async-task-thread-25524] Connection error . Details: org.apache.hc.core5.http.ConnectionRequestTimeoutException: Timeout deadline: 30000 MILLISECONDS, actual: 30000 MILLISECONDS Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.190 ERROR [CompletionServiceHelper] [async-task-thread-25538] Connection error . Details: org.apache.hc.core5.http.ConnectionRequestTimeoutException: Timeout deadline: 30000 MILLISECONDS, actual: 30000 MILLISECONDS Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.198 ERROR [CompletionServiceHelper] [async-task-thread-25510] Connection error . Details: org.apache.hc.core5.http.ConnectionRequestTimeoutException: Timeout deadline: 30000 MILLISECONDS, actual: 30000 MILLISECONDS Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.200 ERROR [CompletionServiceHelper] [async-task-thread-25457] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.200 ERROR [CompletionServiceHelper] [async-task-thread-25434] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.200 ERROR [CompletionServiceHelper] [async-task-thread-25462] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.210 ERROR [CompletionServiceHelper] [async-task-thread-25513] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.210 ERROR [CompletionServiceHelper] [async-task-thread-25459] Connection error . Details: java.net.SocketTimeoutException: Read timed out Sep 16 080556 aiops-5g-pinot-test-controller1 pinot-admin.sh[558773]: 2025/09/16 080556.535 ERROR [CompletionServiceHelper] [async-task-thread-25533] Connection error . Details: org.apache.hc.core5.http.ConnectionRequestTimeoutException: Timeout deadline: 30000 MILLISECONDS, actual: 30000 MILLISECONDS

Luis P Fernandes

09/16/2025, 7:07 AM

Would enabling peer download for the segments help? If so not clear if this works with NFS, do server need direct access to the mount point?

Xiang Fu

09/16/2025, 7:38 AM

server just need local storage access

Xiang Fu

09/16/2025, 7:38 AM

controller needs shared point for deep store

Luis P Fernandes

09/16/2025, 7:47 AM

Is this something that can help for the current situation of reloading a lot of segments ?

Luis P Fernandes

09/18/2025, 9:47 AM

Hi Guys, quick update we manage to recover the cluster by bringing servers up in a sequence. After we applied the rollup and have reduced the number of segments in the cluster. We have renabled live data ingestion, however we still have plenty of errors such as the following: Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: Sep 18, 2025 103131 AM org.glassfish.jersey.server.ServerRuntime$Responder writeResponse Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: SEVERE: An I/O error has occurred while writing a response message entity to the container output stream. Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection is closed Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionW rapperInterceptor.java:67) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:13 9) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:649) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:370) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:259) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.internal.Errors.process(Errors.java:292) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.internal.Errors.process(Errors.java:274) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.internal.Errors.process(Errors.java:244) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/java.lang.Thread.run(Thread.java:1583) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: Caused by: java.io.IOException: Connection is closed Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.grizzly.nio.NIOConnection.assertOpen(NIOConnection.java:420) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:613) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.grizzly.http.server.NIOOutputStreamImpl.write(NIOOutputStreamImpl.java:60) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:203) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInter ceptorExecutor.java:276) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:309) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:381) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:357) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/sun.nio.cs.StreamEncoder.lockedWrite(StreamEncoder.java:158) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:139) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:167) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:237) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at java.base/java.io.Writer.write(Writer.java:278) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:249) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:76) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:36) Sep 18 103131 aiops-5g-pinot-test-server1 pinot-admin.sh[3131393]: at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242) Does anyone has any suggestions for what we should look at to identify the root cause, thanks.

Luis P Fernandes

09/20/2025, 9:45 AM

HI Guys, we still cant get our cluster working properly, we continue to get all sorts of timeouts, IO errors. This are triggered by any kind of queries, the sequence of events seems to be after query is lunched we get a timeout in ZK that than triggers errors on the server and disconnection from ZK leading to ZK session reset. The following is what I see from the logs: Server 2: 2025/09/20 110339.364 WARN [AppInfoParser] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Error registering AppInfo mbean 2025/09/20 111352.118 WARN [AppInfoParser] [HelixTaskExecutor-message_handle_STATE_TRANSITION] Error registering AppInfo mbean 2025/09/20 111837.167 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(100.100.80.80:2191)] Client session timed out, have not heard from server in 130563ms for session id 0x1000b3db7c00135 2025/09/20 111837.167 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(100.100.80.80:2191)] Session 0x1000b3db7c00135 for server aiops-5g-pinot-test-zookeeper1/100.100.80.80:2191, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException. 2025/09/20 112044.564 WARN [ZKHelixManager] [ZkClient-EventThread-126-100.100.80.80:2191] KeeperState:Expired, SessionId: 1000b3db7c00135, instance: Server_pinot-server2_8098, type: PARTICIPANT Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: 2025/09/20 113051.933 ERROR [ServerQueryExecutorV1Impl] [pqr-1] Exception processing requestId 2030915823000000005 Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: java.lang.RuntimeException: Caught exception while running CombinePlanNode. Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.plan.CombinePlanNode.lambda$getCombineOperator$2(CombinePlanNode.java:124) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.util.QueryMultiThreadingUtils.runTasksWithDeadline(QueryMultiThreadingUtils.java:101) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.plan.CombinePlanNode.getCombineOperator(CombinePlanNode.java:104) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:82) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:44) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:51) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.executeQuery(ServerQueryExecutorV1Impl.java:736) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.execute(ServerQueryExecutorV1Impl.java:716) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.executeInternal(ServerQueryExecutorV1Impl.java:448) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.executeInternal(ServerQueryExecutorV1Impl.java:319) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.execute(ServerQueryExecutorV1Impl.java:144) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.query.executor.QueryExecutor.execute(QueryExecutor.java:59) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:142) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:124) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba161 34] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: Caused by: java.util.concurrent.TimeoutException Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204) ~[?:?] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: at org.apache.pinot.core.util.QueryMultiThreadingUtils.runTasksWithDeadline(QueryMultiThreadingUtils.java:97) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] Sep 20 113051 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: ... 20 more Sep 20 113052 aiops-5g-pinot-test-server2 pinot-admin.sh[5271]: 2025/09/20 113052.160 ERROR [RealtimeSegmentDataManager_elastic_system_filesystem__0__261__20250920T0710Z] [elastic_system_filesystem__0__261__20250920T0710Z] Message loss detected in stream partition: 0 for table: elastic_system_filesystem_REALTIME startOffset: 630023097 batchFirstOffset: 630659381 ZK: Sep 20 111740 aiops-5g-pinot-test-zookeeper1 zkServer.sh[126993]: 2025-09-20 111740,554 [myid:] - INFO [SessionTracker:o.a.z.s.ZooKeeperServer@730] - Expiring session 0x1000b3db7c00136, timeout of 90000ms exceeded Sep 20 111740 aiops-5g-pinot-test-zookeeper1 zkServer.sh[126993]: 2025-09-20 111740,554 [myid:] - INFO [RequestThrottler:o.a.z.s.q.QuorumZooKeeperServer@156] - Submitting global closeSession request for session 0x1000b3db7c00136 Sep 20 111752 aiops-5g-pinot-test-zookeeper1 zkServer.sh[126993]: 2025-09-20 111752,554 [myid:] - INFO [SessionTracker:o.a.z.s.ZooKeeperServer@730] - Expiring session 0x1000b3db7c0013e, timeout of 90000ms exceeded Sep 20 111752 aiops-5g-pinot-test-zookeeper1 zkServer.sh[126993]: 2025-09-20 111752,554 [myid:] - INFO [RequestThrottler:o.a.z.s.q.QuorumZooKeeperServer@156] - Submitting global closeSession request for session 0x1000b3db7c0013e Sep 20 111758 aiops-5g-pinot-test-zookeeper1 zkServer.sh[126993]: 2025-09-20 111758,554 [myid:] - INFO [SessionTracker:o.a.z.s.ZooKeeperServer@730] - Expiring session 0x1000b3db7c00135, timeout of 90000ms exceeded Sep 20 111758 aiops-5g-pinot-test-zookeeper1 zkServer.sh[126993]: 2025-09-20 111758,554 [myid:] - INFO [RequestThrottler:o.a.z.s.q.QuorumZooKeeperServer@156] - Submitting global closeSession request for session 0x1000b3db7c00135 Sep 20 111822 aiops-5g-pinot-test-zookeeper1 zkServer.sh[126993]: 2025-09-20 111822,554 [myid:] - INFO [SessionTracker:o.a.z.s.ZooKeeperServer@730] - Expiring session 0x1000b3db7c00141, timeout of 90000ms exceeded Controller: 2025/09/20 111746.452 WARN [ZKHelixManager] [pool-1-thread-4] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 111801.448 WARN [ZKHelixManager] [pool-1-thread-5] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 111816.445 WARN [ZKHelixManager] [pool-1-thread-1] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 111831.449 WARN [ZKHelixManager] [pool-1-thread-2] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 111846.457 WARN [ZKHelixManager] [pool-1-thread-3] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 111901.463 WARN [ZKHelixManager] [pool-1-thread-4] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 111916.443 WARN [ZKHelixManager] [pool-1-thread-5] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 111931.447 WARN [ZKHelixManager] [pool-1-thread-1] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 111946.451 WARN [ZKHelixManager] [pool-1-thread-2] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 112001.460 WARN [ZKHelixManager] [pool-1-thread-3] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 112016.451 WARN [ZKHelixManager] [pool-1-thread-4] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 112031.456 WARN [ZKHelixManager] [pool-1-thread-5] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010 2025/09/20 112046.462 WARN [ZKHelixManager] [pool-1-thread-1] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010

Xiang Fu

09/20/2025, 11:26 AM

this error means the data in kafka is already lost

Message loss detected in stream partition: 0 for table: elastic_system_filesystem_REALTIME startOffset: 630023097 batchFirstOffset: 630659381

you can pause the ingestion and resume from latest

Xiang Fu

09/20/2025, 11:27 AM

this warning:

WARN [ZKHelixManager] [pool-1-thread-4] Instance pinot-controller1_9000 is not leader of cluster cno_telemetry_pinot due to current session 1000b3db7c0006d does not match leader session 1000b3db7c00010

should go away after zookeeper znode timeout

Open in Slack

Previous Next