Hi Team,
We have set up a multi-node Pinot cluster with 3 controller nodes on EKS. I have a table with a replication factor of 2, and we can see that each segment is distributed between two of the three server nodes. However, when we check the segment download URL, all of them point to the same controller node. The data directories on the other two controller nodes do not contain any segments. Is this the expected behavior?
Please let me know if you need any additional information to determine if any configuration changes are required. Below controller is pointed for all segment download url's
> kubectl describe pod pinot-controller-2 -n pinot
Name: pinot-controller-2
Namespace: pinot
Priority: 0
Service Account: default
Node: ip-
private-ip.ap-south-1.compute.internal/
Start Time: Fri, 07 Mar 2025 13
2745 +0530
Labels: app=pinot
apps.kubernetes.io/pod-index=2
component=controller
controller-revision-hash=pinot-controller-8fc9c98c9
statefulset.kubernetes.io/pod-name=pinot-controller-2
Annotations:
kubectl.kubernetes.io/restartedAt: 2025-03-07T13
2606+05:30
Status: Running
IP:
ip
IPs:
IP:
ip
Controlled By: StatefulSet/pinot-controller
Containers:
pinot-controller:
Container ID:
containerd://b7eee5b7ad36e6b532e5005e1292c86305f756ddbf748bdd17d297c75b500b0d
Image: apachepinot/pinot:1.1.0
Image ID:
docker.io/apachepinot/pinot@sha256:84c50e845fade3d00809decca4daeb658d9347628e4e564b707cadf32dd3b74f
Port: 9000/TCP
Host Port: 0/TCP
Command:
/opt/pinot/bin/pinot-admin.sh
StartController
-zkAddress
zookeeper-0.
hostname.local:2181,zookeeper-1.
hostname.local:2181,zookeeper-2.
hostname.local:2181
-controllerHost
$(POD_NAME).
hostname.local
-clusterName
PinotCluster
State: Running
Started: Mon, 24 Mar 2025 20
1730 +0530
Last State: Terminated
Reason: Error
Exit Code: 143
Started: Mon, 24 Mar 2025 19
4220 +0530
Finished: Mon, 24 Mar 2025 20
1730 +0530
Ready: True
Restart Count: 5
Environment:
POD_NAME: pinot-controller-2 (v1:metadata.name)
JAVA_OPTS: -Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200
Mounts:
/opt/pinot/logs from controller-logs (rw)
/tmp/data from controller-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7rfw5 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
controller-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: controller-data-pinot-controller-2
ReadOnly: false
controller-logs:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: controller-logs-pinot-controller-2
ReadOnly: false
kube-api-access-7rfw5:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
nil
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
none
Tolerations:
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 6m10s (x6 over 17d) kubelet Container image "apachepinot/pinot:1.1.0" already present on machine
Normal Created 6m10s (x6 over 17d) kubelet Created container pinot-controller
Normal Started 6m10s (x6 over 17d) kubelet Started container pinot-controller