Hi I have followed this documentation to query pinot using J apache-pinot #troubleshooting

Hi, I have followed this documentation to query pinot using Java client: https://docs.pinot.apache.org/users/clients/java I have scheduled the query to run after every 24 hours. The first run goes fine. However the subsequent iteration fails with the below exception. 2021-06-15 052309.880 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052309.910 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) 2021-06-15 052309.910 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181, initiating session 2021-06-15 052309.911 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181, initiating session 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052310.279 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) Any help here?

Kishore G

06/15/2021, 6:47 AM

its probably GCing

whats the query? is it pulling a lot of data out of Pinot?

khush

06/15/2021, 6:48 AM

There is no data currently

Hardly couple of rows

Kishore G

06/15/2021, 6:50 AM

running in k8s?

khush

06/15/2021, 6:50 AM

Yes

Kishore G

06/15/2021, 6:50 AM

can you check the pod status and vm resources

most likely pods are crashing

khush

06/15/2021, 6:50 AM

No pod isn't crashing

Its running fine

There is connection leakage from the report service, which queries pinot. Steps taken to figure this out: 1. Checked pinot zookeeper pod logs on k8s 2. Post that used the following command to get all the connection details for all clients to this server: echo stat | nc <zk_host> <zk port> echo cons | nc <zk_host> <zk port> All the connection was identified to be from a single ip which belonged to the report service.

Kishore G

06/15/2021, 7:34 AM

👍

khush

06/15/2021, 7:43 AM

@Kishore G thank you for the quick help