Hi,
I have followed this documentation to query pi...
# troubleshootingk
khush
06/15/2021, 6:23 AMHi,
I have followed this documentation to query pinot using Java client: https://docs.pinot.apache.org/users/clients/java
I have scheduled the query to run after every 24 hours. The first run goes fine. However the subsequent iteration fails with the below exception.
2021-06-15 052309.880 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2021-06-15 052309.910 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error)
2021-06-15 052309.910 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181, initiating session
2021-06-15 052309.911 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error)
2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Socket connection established to pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181, initiating session
2021-06-15 052309.941 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2021-06-15 052310.279 INFO 11 --- [ter.local:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server pinot2-zookeeper.data2.svc.cluster.local/10.100.71.32:2181. Will not attempt to authenticate using SASL (unknown error)
Any help here?
k
Kishore G
06/15/2021, 6:47 AM
its probably GCing
Kishore G
06/15/2021, 6:47 AM
whats the query? is it pulling a lot of data out of Pinot?
k
khush
06/15/2021, 6:48 AMThere is no data currently
khush
06/15/2021, 6:48 AMHardly couple of rows
k
Kishore G
06/15/2021, 6:50 AM
running in k8s?
k
khush
06/15/2021, 6:50 AMYes
k
Kishore G
06/15/2021, 6:50 AM
can you check the pod status and vm resources
Kishore G
06/15/2021, 6:50 AM
most likely pods are crashing
k
khush
06/15/2021, 6:50 AMNo pod isn't crashing
khush
06/15/2021, 6:50 AMIts running fine
khush
06/15/2021, 7:31 AMThere is connection leakage from the report service, which queries pinot. Steps taken to figure this out:
1. Checked pinot zookeeper pod logs on k8s
2. Post that used the following command to get all the connection details for all clients to this server:
echo stat | nc <zk_host> <zk port>
echo cons | nc <zk_host> <zk port>
All the connection was identified to be from a single ip which belonged to the report service.
k
Kishore G
06/15/2021, 7:34 AM
👍
k
khush
06/15/2021, 7:43 AM@Kishore G thank you for the quick help