Hi Team, We have set up a multi-node Pinot cluster...
# troubleshooting
p
Hi Team, We have set up a multi-node Pinot cluster with 3 controller nodes on EKS. I have a table with a replication factor of 2, and we can see that each segment is distributed between two of the three server nodes. However, when we check the segment download URL, all of them point to the same controller node. The data directories on the other two controller nodes do not contain any segments. Is this the expected behavior? Please let me know if you need any additional information to determine if any configuration changes are required. Below controller is pointed for all segment download url's > kubectl describe pod pinot-controller-2 -n pinot Name: pinot-controller-2 Namespace: pinot Priority: 0 Service Account: default Node: ip-private-ip.ap-south-1.compute.internal/ Start Time: Fri, 07 Mar 2025 132745 +0530 Labels: app=pinot apps.kubernetes.io/pod-index=2 component=controller controller-revision-hash=pinot-controller-8fc9c98c9 statefulset.kubernetes.io/pod-name=pinot-controller-2 Annotations: kubectl.kubernetes.io/restartedAt: 2025-03-07T132606+05:30 Status: Running IP: ip IPs: IP: ip Controlled By: StatefulSet/pinot-controller Containers: pinot-controller: Container ID: containerd://b7eee5b7ad36e6b532e5005e1292c86305f756ddbf748bdd17d297c75b500b0d Image: apachepinot/pinot:1.1.0 Image ID: docker.io/apachepinot/pinot@sha256:84c50e845fade3d00809decca4daeb658d9347628e4e564b707cadf32dd3b74f Port: 9000/TCP Host Port: 0/TCP Command: /opt/pinot/bin/pinot-admin.sh StartController -zkAddress zookeeper-0.hostname.local:2181,zookeeper-1.hostname.local:2181,zookeeper-2.hostname.local:2181 -controllerHost $(POD_NAME).hostname.local -clusterName PinotCluster State: Running Started: Mon, 24 Mar 2025 201730 +0530 Last State: Terminated Reason: Error Exit Code: 143 Started: Mon, 24 Mar 2025 194220 +0530 Finished: Mon, 24 Mar 2025 201730 +0530 Ready: True Restart Count: 5 Environment: POD_NAME: pinot-controller-2 (v1:metadata.name) JAVA_OPTS: -Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 Mounts: /opt/pinot/logs from controller-logs (rw) /tmp/data from controller-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7rfw5 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: controller-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: controller-data-pinot-controller-2 ReadOnly: false controller-logs: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: controller-logs-pinot-controller-2 ReadOnly: false kube-api-access-7rfw5: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: nil DownwardAPI: true QoS Class: BestEffort Node-Selectors: none Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 6m10s (x6 over 17d) kubelet Container image "apachepinot/pinot:1.1.0" already present on machine Normal Created 6m10s (x6 over 17d) kubelet Created container pinot-controller Normal Started 6m10s (x6 over 17d) kubelet Started container pinot-controller
x
What’s the download url looks like? If you are on EKS, try to setup s3 as your deep store
p
Thanks for your response ....We will give a try