we have drilled down the issue, reached out to aws...
# troubleshooting
a
we have drilled down the issue, reached out to aws premium support team, and this is the report. Hello, Greetings of the day! Thank you contacting AWS Premium Support. Hope you are doing well. This is Ajay from AWS Containers team. We had connected through a chat session followed by chimes call, I am writing this email to provide you a summary of our conversation with regard to this case. ISSUE: ------------ You reached out to troubleshoot the Apache pinot as it failed to start in eks SUMMARY: ------------ I have replicated the scenario and can found the same situation where all pods of pinot gets either Crashloopbackoff status or ContainerCreating status. I have inspected the logs of the following pods and could see the application is failing to eastablish connections with Zookeeper. Sharing you below my replication results. [+] Have pulled the repo using below commands as mentioned in Apache pinot documentation .[1] # checkout pinot git clone https://github.com/apache/pinot.git cd pinot/helm/pinot Updated Helm dependency helm dependency update From below path in the above repo macpc@test helm % pwd /Users/macpc/eks/pinot/pinot/helm Started Pinot with Helm kubectl create ns pinot-quickstart helm install -n pinot-quickstart pinot ./pinot Created namespace pinot-quickstart macpc@test helm % kubectl get ns NAME STATUS AGE default Active 28h kube-system Active 28h pinot-quickstart Active 21m macpc@test helm % kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 106m gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer false 21h [+] Checked all pvc , all are in bound status ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ macpc@test helm % kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE pinot-quickstart data-pinot-controller-0 Bound pvc-96b61abf-c78c-43bf-b442-66e132458b76 1Gi RWO gp2 <unset> 16m pinot-quickstart data-pinot-server-0 Bound pvc-270fce91-02c3-45e2-b5f4-96553dd6f778 4Gi RWO gp2 <unset> 16m pinot-quickstart data-pinot-zookeeper-0 Bound pvc-c7c3636d-bf7d-4b31-adcb-3369625d4eff 8Gi RWO gp2 <unset> 16m ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ macpc@test helm % kubectl get all -n pinot-quickstart NAME READY STATUS RESTARTS AGE pod/pinot-broker-0 0/1 CrashLoopBackOff 2 (16s ago) 3m43s pod/pinot-controller-0 0/1 ContainerCreating 0 3m42s pod/pinot-minion-stateless-7964865977-ddh2p 0/1 CrashLoopBackOff 2 (19s ago) 3m44s pod/pinot-server-0 0/1 ContainerCreating 0 3m42s pod/pinot-zookeeper-0 0/1 ContainerCreating 0 3m43s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/pinot-broker ClusterIP 10.100.137.195 <none> 8099/TCP 3m47s service/pinot-broker-headless ClusterIP None <none> 8099/TCP 3m47s service/pinot-controller ClusterIP 10.100.164.135 <none> 9000/TCP 3m45s service/pinot-controller-external LoadBalancer 10.100.129.104 a87d66303484449a29dc659862baf4f6-425728138.us-east-1.elb.amazonaws.com 9000:31960/TCP 3m46s service/pinot-controller-headless ClusterIP None <none> 9000/TCP 3m46s service/pinot-server ClusterIP 10.100.51.155 <none> 8098/TCP,80/TCP 3m44s service/pinot-server-headless ClusterIP None <none> 8098/TCP,80/TCP 3m45s service/pinot-zookeeper ClusterIP 10.100.198.212 <none> 2181/TCP 3m49s service/pinot-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 3m49s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/pinot-minion-stateless 0/1 1 0 3m45s NAME DESIRED CURRENT READY AGE replicaset.apps/pinot-minion-stateless-7964865977 1 1 0 3m45s NAME READY AGE statefulset.apps/pinot-broker 0/1 3m44s statefulset.apps/pinot-controller 0/1 3m43s statefulset.apps/pinot-server 0/1 3m43s statefulset.apps/pinot-zookeeper 0/1 3m44s [+] Checked the logs for the pod pinot-broker-0 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ macpc@test helm % kubectl logs pod/pinot-broker-0 -n pinot-quickstart java.net.ConnectException: Connection refused at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1291) [pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965] 2025/04/08 120837.982 INFO [ClientCnxn] [Start a Pinot [BROKER]-SendThread(pinot-zookeeper:2181)] Opening socket connection to server pinot-zookeeper/10.100.198.212:2181. 2025/04/08 120837.982 INFO [ClientCnxn] [Start a Pinot [BROKER]-SendThread(pinot-zookeeper:2181)] SASL config status: Will not attempt to authenticate using SASL (unknown error) 2025/04/08 120837.982 WARN [ClientCnxn] [Start a Pinot [BROKER]-SendThread(pinot-zookeeper:2181)] Session 0x0 for server pinot-zookeeper/10.100.198.212:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException. java.net.ConnectException: Connection refused at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1291) [pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965] ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ From above logs This error log shows that the Pinot broker is unable to connect to ZooKeeper, which is a critical component for Apache Pinot's cluster coordination. Here's what's happening: The broker is trying to connect to pinot-zookeeper:2181 but getting "Connection refused" errors SASL authentication is not configured (which is normal in many development setups) The connection keeps failing and attempting to reconnect [+] Describe output for pod pinot-broker-0 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ macpc@test helm % kubectl logs pod/pinot-broker-0 -n pinot-quickstart Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 8m50s default-scheduler Successfully assigned pinot-quickstart/pinot-broker-0 to ip-192-168-18-194.ec2.internal Normal Pulled 8m49s kubelet Successfully pulled image "apachepinot/pinot:latest" in 166ms (166ms including waiting). Image size: 1274580519 bytes. Normal Pulled 7m46s kubelet Successfully pulled image "apachepinot/pinot:latest" in 116ms (116ms including waiting). Image size: 1274580519 bytes. Normal Pulled 6m25s kubelet Successfully pulled image "apachepinot/pinot:latest" in 126ms (126ms including waiting). Image size: 1274580519 bytes. Warning Unhealthy 5m23s kubelet Readiness probe failed: Get "http://192.168.22.221:8099/health": dial tcp 192.168.22.2218099 connect: connection refused Normal Pulling 4m54s (x4 over 8m49s) kubelet Pulling image "apachepinot/pinot:latest" Normal Created 4m54s (x4 over 8m49s) kubelet Created container broker Normal Started 4m54s (x4 over 8m49s) kubelet Started container broker Normal Pulled 4m54s kubelet Successfully pulled image "apachepinot/pinot:latest" in 130ms (130ms including waiting). Image size: 1274580519 bytes. Warning BackOff 3m23s (x9 over 6m43s) kubelet Back-off restarting failed container broker in pod pinot-broker-0_pinot-quickstart(4a110518-8b8e-48a9-a5ef-3635dca53764) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [+] I have tried deleting the pvc data-pinot-zookeeper-0 manually and so does the pod. Though the new pod and pvc have been created and pvc came to bound status , still pod failed to come up in running sttaus. You can verify the pvc age as 19s , which indicates its newly created macpc@test helm % kubectl get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE pinot-quickstart data-pinot-controller-0 Bound pvc-96b61abf-c78c-43bf-b442-66e132458b76 1Gi RWO gp2 <unset> 39m pinot-quickstart data-pinot-server-0 Bound pvc-270fce91-02c3-45e2-b5f4-96553dd6f778 4Gi RWO gp2 <unset> 39m pinot-quickstart data-pinot-zookeeper-0 Bound pvc-22477e27-ba57-4a4b-8e35-6c73111e6f7d 8Gi RWO gp2 <unset> 19s I would suggest you to get in touch with Apache team as the pod is failing to run , though there is no error shown in pod describe output. Apache pinot is a third party tool and it comes out of AWS Support scope, hence a deep troubleshooting wont be possible.
x
The sequence for pinot to start is zookeeper -> controller -> other components
so please first check why zookeeper is not up.
a
can we connect over a call to check thid
even zookeeper shows the same issues
Screenshot 2025-04-08 at 8.48.41 PM.png
x
then need to fix this zookeeper binding thing first
which storage class is this
can you please check if PV is created?
And try to use gp3 for the storage for all pinot components
a
we have tried using gp3, pv was checked too
x
that’s weird
is this resolved?