https://pinot.apache.org/ logo
k

khush

06/15/2021, 6:23 AM
Hi, I have followed this documentation to query pinot using Java client: https://docs.pinot.apache.org/users/clients/java I have scheduled the query to run after every 24 hours. The first run goes fine. However the subsequent iteration fails with the below exception. 2021-06-15 052309.880 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052309.910 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) 2021-06-15 052309.910 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181, initiating session 2021-06-15 052309.911 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181, initiating session 2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2021-06-15 052310.279 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error) Any help here?
k

Kishore G

06/15/2021, 6:47 AM
its probably GCing
whats the query? is it pulling a lot of data out of Pinot?
k

khush

06/15/2021, 6:48 AM
There is no data currently
Hardly couple of rows
k

Kishore G

06/15/2021, 6:50 AM
running in k8s?
k

khush

06/15/2021, 6:50 AM
Yes
k

Kishore G

06/15/2021, 6:50 AM
can you check the pod status and vm resources
most likely pods are crashing
k

khush

06/15/2021, 6:50 AM
No pod isn't crashing
Its running fine
There is connection leakage from the report service, which queries pinot. Steps taken to figure this out: 1. Checked pinot zookeeper pod logs on k8s 2. Post that used the following command to get all the connection details for all clients to this server: echo stat | nc <zk_host> <zk port> echo cons | nc <zk_host> <zk port> All the connection was identified to be from a single ip which belonged to the report service.
k

Kishore G

06/15/2021, 7:34 AM
👍
k

khush

06/15/2021, 7:43 AM
@Kishore G thank you for the quick help