we have drilled down the issue, reached out to aws premium support team, and this is the report. Hello,
Greetings of the day!
Thank you contacting AWS Premium Support. Hope you are doing well. This is Ajay from AWS Containers team.
We had connected through a chat session followed by chimes call, I am writing this email to provide you a summary of our conversation with regard to this case.
ISSUE:
------------
You reached out to troubleshoot the Apache pinot as it failed to start in eks
SUMMARY:
------------
I have replicated the scenario and can found the same situation where all pods of pinot gets either Crashloopbackoff status or ContainerCreating status. I have inspected the logs of the following pods and could see the application is failing to eastablish connections with Zookeeper.
Sharing you below my replication results.
[+] Have pulled the repo using below commands as mentioned in Apache pinot documentation .[1]
# checkout pinot
git clone
https://github.com/apache/pinot.git
cd pinot/helm/pinot
Updated Helm dependency
helm dependency update
From below path in the above repo
macpc@test helm % pwd
/Users/macpc/eks/pinot/pinot/helm
Started Pinot with Helm
kubectl create ns pinot-quickstart
helm install -n pinot-quickstart pinot ./pinot
Created namespace pinot-quickstart
macpc@test helm % kubectl get ns
NAME STATUS AGE
default Active 28h
kube-system Active 28h
pinot-quickstart Active 21m
macpc@test helm % kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
gp2 (default)
ebs.csi.aws.com Delete WaitForFirstConsumer true 106m
gp3 (default)
ebs.csi.aws.com Delete WaitForFirstConsumer false 21h
[+] Checked all pvc , all are in bound status
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
macpc@test helm % kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
pinot-quickstart data-pinot-controller-0 Bound pvc-96b61abf-c78c-43bf-b442-66e132458b76 1Gi RWO gp2 <unset> 16m
pinot-quickstart data-pinot-server-0 Bound pvc-270fce91-02c3-45e2-b5f4-96553dd6f778 4Gi RWO gp2 <unset> 16m
pinot-quickstart data-pinot-zookeeper-0 Bound pvc-c7c3636d-bf7d-4b31-adcb-3369625d4eff 8Gi RWO gp2 <unset> 16m
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
macpc@test helm % kubectl get all -n pinot-quickstart
NAME READY STATUS RESTARTS AGE
pod/pinot-broker-0 0/1 CrashLoopBackOff 2 (16s ago) 3m43s
pod/pinot-controller-0 0/1 ContainerCreating 0 3m42s
pod/pinot-minion-stateless-7964865977-ddh2p 0/1 CrashLoopBackOff 2 (19s ago) 3m44s
pod/pinot-server-0 0/1 ContainerCreating 0 3m42s
pod/pinot-zookeeper-0 0/1 ContainerCreating 0 3m43s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/pinot-broker ClusterIP 10.100.137.195 <none> 8099/TCP 3m47s
service/pinot-broker-headless ClusterIP None <none> 8099/TCP 3m47s
service/pinot-controller ClusterIP 10.100.164.135 <none> 9000/TCP 3m45s
service/pinot-controller-external LoadBalancer 10.100.129.104
a87d66303484449a29dc659862baf4f6-425728138.us-east-1.elb.amazonaws.com 9000:31960/TCP 3m46s
service/pinot-controller-headless ClusterIP None <none> 9000/TCP 3m46s
service/pinot-server ClusterIP 10.100.51.155 <none> 8098/TCP,80/TCP 3m44s
service/pinot-server-headless ClusterIP None <none> 8098/TCP,80/TCP 3m45s
service/pinot-zookeeper ClusterIP 10.100.198.212 <none> 2181/TCP 3m49s
service/pinot-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 3m49s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/pinot-minion-stateless 0/1 1 0 3m45s
NAME DESIRED CURRENT READY AGE
replicaset.apps/pinot-minion-stateless-7964865977 1 1 0 3m45s
NAME READY AGE
statefulset.apps/pinot-broker 0/1 3m44s
statefulset.apps/pinot-controller 0/1 3m43s
statefulset.apps/pinot-server 0/1 3m43s
statefulset.apps/pinot-zookeeper 0/1 3m44s
[+] Checked the logs for the pod pinot-broker-0
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
macpc@test helm % kubectl logs pod/pinot-broker-0 -n pinot-quickstart
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1291) [pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
2025/04/08 12
0837.982 INFO [ClientCnxn] [Start a Pinot [BROKER]-SendThread(pinot-zookeeper:2181)] Opening socket connection to server pinot-zookeeper/10.100.198.212:2181.
2025/04/08 12
0837.982 INFO [ClientCnxn] [Start a Pinot [BROKER]-SendThread(pinot-zookeeper:2181)] SASL config status: Will not attempt to authenticate using SASL (unknown error)
2025/04/08 12
0837.982 WARN [ClientCnxn] [Start a Pinot [BROKER]-SendThread(pinot-zookeeper:2181)] Session 0x0 for server pinot-zookeeper/10.100.198.212:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException.
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) ~[?:?]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1291) [pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
From above logs This error log shows that the Pinot broker is unable to connect to ZooKeeper, which is a critical component for Apache Pinot's cluster coordination. Here's what's happening:
The broker is trying to connect to pinot-zookeeper:2181 but getting "Connection refused" errors
SASL authentication is not configured (which is normal in many development setups)
The connection keeps failing and attempting to reconnect
[+] Describe output for pod pinot-broker-0
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
macpc@test helm % kubectl logs pod/pinot-broker-0 -n pinot-quickstart
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m50s default-scheduler Successfully assigned pinot-quickstart/pinot-broker-0 to ip-192-168-18-194.ec2.internal
Normal Pulled 8m49s kubelet Successfully pulled image "apachepinot/pinot:latest" in 166ms (166ms including waiting). Image size: 1274580519 bytes.
Normal Pulled 7m46s kubelet Successfully pulled image "apachepinot/pinot:latest" in 116ms (116ms including waiting). Image size: 1274580519 bytes.
Normal Pulled 6m25s kubelet Successfully pulled image "apachepinot/pinot:latest" in 126ms (126ms including waiting). Image size: 1274580519 bytes.
Warning Unhealthy 5m23s kubelet Readiness probe failed: Get "
http://192.168.22.221:8099/health": dial tcp 192.168.22.221
8099 connect: connection refused
Normal Pulling 4m54s (x4 over 8m49s) kubelet Pulling image "apachepinot/pinot:latest"
Normal Created 4m54s (x4 over 8m49s) kubelet Created container broker
Normal Started 4m54s (x4 over 8m49s) kubelet Started container broker
Normal Pulled 4m54s kubelet Successfully pulled image "apachepinot/pinot:latest" in 130ms (130ms including waiting). Image size: 1274580519 bytes.
Warning BackOff 3m23s (x9 over 6m43s) kubelet Back-off restarting failed container broker in pod pinot-broker-0_pinot-quickstart(4a110518-8b8e-48a9-a5ef-3635dca53764)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[+] I have tried deleting the pvc data-pinot-zookeeper-0 manually and so does the pod. Though the new pod and pvc have been created and pvc came to bound status , still pod failed to come up in running sttaus.
You can verify the pvc age as 19s , which indicates its newly created
macpc@test helm % kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
pinot-quickstart data-pinot-controller-0 Bound pvc-96b61abf-c78c-43bf-b442-66e132458b76 1Gi RWO gp2 <unset> 39m
pinot-quickstart data-pinot-server-0 Bound pvc-270fce91-02c3-45e2-b5f4-96553dd6f778 4Gi RWO gp2 <unset> 39m
pinot-quickstart data-pinot-zookeeper-0 Bound pvc-22477e27-ba57-4a4b-8e35-6c73111e6f7d 8Gi RWO gp2 <unset> 19s
I would suggest you to get in touch with Apache team as the pod is failing to run , though there is no error shown in pod describe output. Apache pinot is a third party tool and it comes out of AWS Support scope, hence a deep troubleshooting wont be possible.