Hello, I was able to successfully implement pinot ...
# generaly
YogeshS
03/25/2024, 11:17 AMHello, I was able to successfully implement pinot with bitnami zookeeper on kubernetes cluster, with 1-way TLS, auth and S3 deep store enabled.
now, im trying to add 2-way TLS to this, and after adding client.auth.enabled=true config to components, they started restarting in a loop. I was not able to get the cause from pod logs, does anyone went throught this, and can help me ? thanks
v
Vik Gamov
03/25/2024, 1:53 PMCould you share
Kubectl describe pod
And
Kubect get events
outputs
y
YogeshS
03/26/2024, 7:33 AMHi @Vik Gamov, here are the command outputs, thank you for checking on this..
Copy code
Name: pinot-controller-0
Namespace: pinot-gs
Priority: 0
Service Account: pinot
Node: k8s-node-5/10.11.0.65
Start Time: Tue, 26 Mar 2024 12:59:25 +0530
Labels: app=pinot
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/version=1.0.0
component=controller
controller-revision-hash=pinot-controller-7dbd85d5fb
helm.sh/chart=pinot-0.2.8-2
heritage=Helm
release=pinot
statefulset.kubernetes.io/pod-name=pinot-controller-0
Annotations: cni.projectcalico.org/containerID: 877eecb382f3a9e19b5cfe45672cd0bc7a259a5922408e2dc8ac41aa9488a502
cni.projectcalico.org/podIP: 10.14.86.217/32
cni.projectcalico.org/podIPs: 10.14.86.217/32
Status: Running
IP: 10.14.86.217
IPs:
IP: 10.14.86.217
Controlled By: StatefulSet/pinot-controller
Containers:
controller:
Container ID: <containerd://e8e37bafc74ec352c97965730fa9fae66eeb783a5eae52a3b17580733d9f372>9
Image: localrepo/pinot:1.2.0
Image ID: localrepo/pinot@sha256:38a842f7fc67a7c2f41243e7b23dcc601fe4f164603ccb432e9d99d7c60a0a0b
Port: 9443/TCP
Host Port: 0/TCP
Args:
StartController
-configFileName
/var/pinot/controller/config/pinot-controller.conf
State: Running
Started: Tue, 26 Mar 2024 12:59:27 +0530
Ready: False
Restart Count: 0
Requests:
memory: 1280Mi
Liveness: http-get https://:9443/health delay=60s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:9443/health delay=60s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
aws-bucket-secret Secret Optional: false
Environment:
JAVA_OPTS: -Xms256M -Xmx1G -Dlog4j2.configurationFile=/opt/pinot/etc/conf/pinot-controller-log4j2.xml -Dplugins.dir=/opt/pinot/plugins
LOG4J_CONSOLE_LEVEL: info
Mounts:
/keystore from keystore (rw)
/var/pinot/controller/config from config (rw)
/var/pinot/controller/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bdgn9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-pinot-controller-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: pinot-controller-config
Optional: false
keystore:
Type: Secret (a volume populated by a Secret)
SecretName: pinot-cert
Optional: false
kube-api-access-bdgn9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 78s default-scheduler Successfully assigned pinot-gs/pinot-controller-0 to k8s-node-5
Normal Pulled 76s kubelet Container image localrepo/pinot:1.2.0" already present on machine
Normal Created 76s kubelet Created container controller
Normal Started 76s kubelet Started container controller
Warning Unhealthy 7s kubelet Readiness probe failed: Get "<https://10.14.86.217:9443/health>": EOFYogeshS
03/26/2024, 7:34 AM Copy code
LAST SEEN TYPE REASON OBJECT MESSAGE
31m Warning Unhealthy pod/pinot-broker-0 Readiness probe failed: Get "<https://10.14.3.135:8443/health>": EOF
6m13s Warning Unhealthy pod/pinot-broker-0 Liveness probe failed: Get "<https://10.14.3.135:8443/health>": EOF
11m Warning BackOff pod/pinot-broker-0 Back-off restarting failed container broker in pod pinot-broker-0_pinot-gs(5ac4095e-4f2a-495d-9631-b669e1b55571)
4m4s Normal Scheduled pod/pinot-broker-0 Successfully assigned pinot-gs/pinot-broker-0 to k8s-node-9
2m33s Normal Pulled pod/pinot-broker-0 Container image "localrepo/pinot:1.2.0" already present on machine
2m33s Normal Created pod/pinot-broker-0 Created container broker
2m33s Normal Started pod/pinot-broker-0 Started container broker
73s Warning Unhealthy pod/pinot-broker-0 Liveness probe failed: Get "<https://10.14.3.135:8443/health>": EOF
73s Warning Unhealthy pod/pinot-broker-0 Readiness probe failed: Get "<https://10.14.3.135:8443/health>": EOF
73s Normal Killing pod/pinot-broker-0 Container broker failed liveness probe, will be restarted
2m32s Warning Unhealthy pod/pinot-broker-0 Readiness probe failed: Get "<https://10.14.3.135:8443/health>": dial tcp 10.14.3.135:8443: connect: connection refused
4m4s Normal SuccessfulCreate statefulset/pinot-broker create Pod pinot-broker-0 in StatefulSet pinot-broker successful
4m4s Warning RecreatingFailedPod statefulset/pinot-broker StatefulSet pinot-gs/pinot-broker is recreating failed Pod pinot-broker-0
4m4s Normal SuccessfulDelete statefulset/pinot-broker delete Pod pinot-broker-0 in StatefulSet pinot-broker successful
50m Normal Pulled pod/pinot-controller-0 Container image "localrepo/pinot:1.2.0" already present on machine
40m Warning Unhealthy pod/pinot-controller-0 Readiness probe failed: Get "<https://10.14.86.247:9443/health>": EOF
15m Warning Unhealthy pod/pinot-controller-0 Liveness probe failed: Get "<https://10.14.86.247:9443/health>": EOF
5m53s Warning BackOff pod/pinot-controller-0 Back-off restarting failed container controller in pod pinot-controller-0_pinot-gs(88c739d7-4d2e-4b30-8d9a-f8582c37066f)
4m14s Normal Scheduled pod/pinot-controller-0 Successfully assigned pinot-gs/pinot-controller-0 to k8s-node-5
2m42s Normal Pulled pod/pinot-controller-0 Container image "localrepo/pinot:1.2.0" already present on machine
2m42s Normal Created pod/pinot-controller-0 Created container controller
2m42s Normal Started pod/pinot-controller-0 Started container controller
73s Warning Unhealthy pod/pinot-controller-0 Readiness probe failed: Get "<https://10.14.86.217:9443/health>": EOF
73s Normal Killing pod/pinot-controller-0 Container controller failed liveness probe, will be restarted
2m41s Warning Unhealthy pod/pinot-controller-0 Readiness probe failed: Get "<https://10.14.86.217:9443/health>": dial tcp 10.14.86.217:9443: connect: connection refused
4m14s Normal SuccessfulCreate statefulset/pinot-controller create Pod pinot-controller-0 in StatefulSet pinot-controller successful
4m14s Warning RecreatingFailedPod statefulset/pinot-controller StatefulSet pinot-gs/pinot-controller is recreating failed Pod pinot-controller-0
4m14s Normal SuccessfulDelete statefulset/pinot-controller delete Pod pinot-controller-0 in StatefulSet pinot-controller successful
4m8s Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
4m10s Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
4m10s Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
4m10s Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
4m9s Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
4m10s Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
4m10s Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
4m10s Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
52m Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
47m Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
32m Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
12m Normal AddedOrUpdated ingress/pinot-controller Configuration for pinot-gs/pinot-controller was added or updated
21m Warning Unhealthy pod/pinot-minion-stateless-846ddf48b8-gsnw9 Readiness probe failed: Get "<http://10.14.86.217:9514/health>": dial tcp 10.14.86.217:9514: connect: connection refused
11m Warning Unhealthy pod/pinot-minion-stateless-846ddf48b8-gsnw9 Liveness probe failed: Get "<http://10.14.86.217:9514/health>": dial tcp 10.14.86.217:9514: connect: connection refused
6m44s Warning BackOff pod/pinot-minion-stateless-846ddf48b8-gsnw9 Back-off restarting failed container minion-stateless in pod pinot-minion-stateless-846ddf48b8-gsnw9_pinot-gs(b0600be1-9a9f-4ea3-aca6-3e2499cdd1c1)
4m16s Normal Scheduled pod/pinot-minion-stateless-846ddf48b8-qcfgp Successfully assigned pinot-gs/pinot-minion-stateless-846ddf48b8-qcfgp to k8s-node-5
2m45s Normal Pulled pod/pinot-minion-stateless-846ddf48b8-qcfgp Container image "artifactory.build.dlt/core-images/pinot:1.2.0" already present on machine
2m45s Normal Created pod/pinot-minion-stateless-846ddf48b8-qcfgp Created container minion-stateless
2m44s Normal Started pod/pinot-minion-stateless-846ddf48b8-qcfgp Started container minion-stateless
75s Warning Unhealthy pod/pinot-minion-stateless-846ddf48b8-qcfgp Readiness probe failed: Get "<http://10.14.86.136:9514/health>": dial tcp 10.14.86.136:9514: connect: connection refused
75s Warning Unhealthy pod/pinot-minion-stateless-846ddf48b8-qcfgp Liveness probe failed: Get "<http://10.14.86.136:9514/health>": dial tcp 10.14.86.136:9514: connect: connection refused
75s Normal Killing pod/pinot-minion-stateless-846ddf48b8-qcfgp Container minion-stateless failed liveness probe, will be restarted
4m16s Normal SuccessfulCreate replicaset/pinot-minion-stateless-846ddf48b8 Created pod: pinot-minion-stateless-846ddf48b8-qcfgp
6m32s Warning Unhealthy pod/pinot-server-0 Readiness probe failed: Get "<https://10.14.86.136:7443/health>": EOF
11m Warning BackOff pod/pinot-server-0 Back-off restarting failed container server in pod pinot-server-0_pinot-gs(7480e2a0-c12b-4c48-bf18-1a013722ebb2)
4m15s Normal Scheduled pod/pinot-server-0 Successfully assigned pinot-gs/pinot-server-0 to k8s-node-5
2m25s Normal Pulled pod/pinot-server-0 Container image "artifactory.build.dlt/core-images/pinot:1.2.0" already present on machine
2m25s Normal Created pod/pinot-server-0 Created container server
2m25s Normal Started pod/pinot-server-0 Started container server
53s Warning Unhealthy pod/pinot-server-0 Readiness probe failed: Get "<https://10.14.86.247:7443/health>": EOF
63s Normal Killing pod/pinot-server-0 Container server failed liveness probe, will be restarted
4m15s Normal SuccessfulCreate statefulset/pinot-server create Pod pinot-server-0 in StatefulSet pinot-server successful
4m15s Warning RecreatingFailedPod statefulset/pinot-server StatefulSet pinot-gs/pinot-server is recreating failed Pod pinot-server-0
4m15s Normal SuccessfulDelete statefulset/pinot-server delete Pod pinot-server-0 in StatefulSet pinot-server successful
4m15s Warning FailedDelete statefulset/pinot-server delete Pod pinot-server-0 in StatefulSet pinot-server failed error: pods "pinot-server-0" not found