Hi All I need help with kubernetes installation I helm insta Apache Pinot #troubleshooting

Hi All, I need help with kubernetes installation. ...

Fritz Budiyanto

07/03/2021, 6:59 PM

Hi All, I need help with kubernetes installation. I helm install pinot using the chart included in main pinot github (by cloning the entire pinot source code). Apparently broker-0 could not able to come up because of

Cluster structure is not set up for cluster: pinot-quickstart

What went wrong here ?

Copy code

cd incubator-pinot/kubernetes/helm/pinot
helm install -n pinot-quickstart2 pinot .

Copy code

Session establishment complete on server pinot-zookeeper/172.20.47.166:2181, sessionid = 0x1012fddef2b0003, negotiated timeout = 30000
zookeeper state changed (SyncConnected)
MBean HelixZkClient:Key=pinot-quickstart.Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099,Type=SPECTATOR has been registered.
MBean HelixZkClient:Key=pinot-quickstart.Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099,PATH=Root,Type=SPECTATOR has been registered.
KeeperState: SyncConnected, instance: Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099, type: SPECTATOR
Handle new session, instance: Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099, type: SPECTATOR
Handling new session, session id: 1012fddef2b0003, instance: Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099, instanceTye: SPECTATOR, cluster: pinot-quickstart
fail to createClient.
org.apache.helix.HelixException: Cluster structure is not set up for cluster: pinot-quickstart
        at org.apache.helix.manager.zk.ZKHelixManager.handleNewSession(ZKHelixManager.java:1124) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-47a75e5093129cc280de4c118434ccb337cd3da1]

Fritz Budiyanto

07/03/2021, 8:20 PM

I found the issue. I'm using AWS EKS, and the default storageClass is "gp2". The helm chart storageClass was set to "" and it didn't choose the default nor map to "gp2"

Xiang Fu

07/03/2021, 8:30 PM

typically start order is: zookeeper/controller/(broker/servers/minion)

Xiang Fu

07/03/2021, 8:30 PM

i thought the storageclass should be default?

Xiang Fu

07/03/2021, 8:31 PM

which points to your system default one

Xiang Fu

07/03/2021, 8:31 PM

e.g.

Copy code

➜ kt get storageclass
NAME                PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
azurefile           <http://kubernetes.io/azure-file|kubernetes.io/azure-file>   Delete          Immediate              true                   57d
azurefile-premium   <http://kubernetes.io/azure-file|kubernetes.io/azure-file>   Delete          Immediate              true                   57d
default (default)   <http://kubernetes.io/azure-disk|kubernetes.io/azure-disk>   Delete          WaitForFirstConsumer   true                   57d
managed-premium     <http://kubernetes.io/azure-disk|kubernetes.io/azure-disk>   Delete          WaitForFirstConsumer   true                   57d

Fritz Budiyanto

07/03/2021, 8:31 PM

Looks like i have 2 defaults:

Copy code

Warning  FailedCreate  5m20s (x21 over 46m)  statefulset-controller  create Claim data-pinot-controller-0 for Pod pinot-controller-0 in StatefulSet pinot-controller failed error: persistentvolumeclaims "data-pinot-controller-0" is forbidden: Internal error occurred: 2 default StorageClasses were found
  Warning  FailedCreate  5m20s (x21 over 46m)  statefulset-controller  create Pod pinot-controller-0 in StatefulSet pinot-controller failed error: failed to create PVC data-pinot-controller-0: persistentvolumeclaims "data-pinot-controller-0" is forbidden: Internal error occurred: 2 default StorageClasses were found

Xiang Fu

07/03/2021, 8:31 PM

oic

Xiang Fu

07/03/2021, 8:31 PM

then this will fail

Xiang Fu

07/03/2021, 8:31 PM

you nee to configure it explicitly

Fritz Budiyanto

07/03/2021, 8:32 PM

Yes thanks. What would be a good storage for K8S deployment ?

Fritz Budiyanto

07/03/2021, 8:32 PM

Would you recommend EBS ?

Xiang Fu

07/03/2021, 8:32 PM

typically for controller you can use s3, server you can use ebs

Xiang Fu

07/03/2021, 8:32 PM

for testing you can use ebs for both

Xiang Fu

07/03/2021, 8:33 PM

controller should have access to deep store, which can be mounted ebs or s3

Fritz Budiyanto

07/03/2021, 8:34 PM

Thanks. I'm planning to use S3 for deep storage as well

Xiang Fu

07/03/2021, 8:34 PM

servers will talk to deep store through controller api(for mounted ebs cases) or s3 api

Xiang Fu

07/03/2021, 8:34 PM

see: https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot

Ryan Clark

07/06/2021, 5:06 PM

I have this same problem. I changed

values.yml

to give

storageClass

"gp2"

. It didn't work. Is this what you mean to configure it explicitly? Or do I need to change the

/templates

Copy code

values.yaml
76:    storageClass: "gp2"
244:    storageClass: "gp2"
245:    #storageClass: "ssd"
330:    storageClass: "gp2"
331:    #storageClass: "ssd"

Ryan Clark

07/06/2021, 5:42 PM

also, gp2 is my default:

Copy code

% kubectl get storageclass
NAME            PROVISIONER             AGE
gp2 (default)   <http://kubernetes.io/aws-ebs|kubernetes.io/aws-ebs>   143m

Xiang Fu

07/06/2021, 7:02 PM

where does it stop?

Xiang Fu

07/06/2021, 7:02 PM

can you describe the pod and check

Xiang Fu

07/06/2021, 7:02 PM

also check if PVCs are already created

Ryan Clark

07/06/2021, 7:05 PM

4 pods are in error. pinot-broker-0:

org.apache.helix.HelixException: Cluster structure is not set up for cluster: pinot-quickstart

pinot-controller-0:

Socket error occurred: localhost/127.0.0.1:2181: Connection refused

pinot-minion-0:

org.apache.helix.HelixException: Cluster structure is not set up for cluster: pinot-quickstart

pinot-server-0:

org.apache.helix.HelixException: Cluster structure is not set up for cluster: pinot-quickstart

Ryan Clark

07/06/2021, 7:07 PM

PVC was created- I can see it in my AWS console.

Ryan Clark

07/06/2021, 7:08 PM

I guess the pinot-quickstart-standard-workers-Node
EBS is gp3. I don't know if that matters

Ryan Clark

07/06/2021, 7:39 PM

I even get these errors running this on minikube

Xiang Fu

07/06/2021, 9:09 PM

I see, is zookeeper pod up?

Xiang Fu

07/06/2021, 9:10 PM

The startup sequence should be zookeeper->controller ->broker/server

Ryan Clark

07/06/2021, 9:12 PM

Zookeeper is up. The pods seem to start in that order.

Fritz Budiyanto

07/06/2021, 9:14 PM

Socket error occurred: localhost/127.0.0.1:2181: Connection refused

seemed like controller connection to ZK had issue.

Ryan Clark

07/06/2021, 9:32 PM

@Fritz Budiyanto your error did not involve the controller refusing to connect?

Fritz Budiyanto

07/06/2021, 9:34 PM

No, connection to ZK was okay. You can double check by manually connecting to ZK with zkCli.sh, comes with ZK distribution

🙌 1

Fritz Budiyanto

07/06/2021, 9:34 PM

ZK also has storageClass. You may want to check if ZK came up okay.

Ryan Clark

07/06/2021, 9:35 PM

How did you explicitly define storageClass? Just in the

values.yaml

Fritz Budiyanto

07/06/2021, 9:36 PM

Copy code

zookeeper:
  ## If true, install the Zookeeper chart alongside Pinot
  ## ref: <https://github.com/kubernetes/charts/tree/master/incubator/zookeeper>
  enabled: true
  storageClass: "efs-sc-dyn"

Fritz Budiyanto

07/06/2021, 9:36 PM

I was using EFS for ZK

Xiang Fu

07/06/2021, 10:02 PM

hmm, why it’s connecting to localhost:2181?

Xiang Fu

07/06/2021, 10:02 PM

it should be

zookeeper:2181

Ryan Clark

07/06/2021, 10:03 PM

I have no idea- I'm really just following the quickstart instructions. Would it help if i shared a video of my entire quickstart?

Xiang Fu

07/06/2021, 10:07 PM

hmm, which quickstart?

Xiang Fu

07/06/2021, 10:07 PM

the helmchart one ?

Ryan Clark

07/06/2021, 10:07 PM

yeah

Xiang Fu

07/06/2021, 10:08 PM

have you modified the zookeeper in values.yaml ?

👎 1

Ryan Clark

07/06/2021, 10:08 PM

Nope

Xiang Fu

07/06/2021, 10:10 PM

i see

Xiang Fu

07/06/2021, 10:15 PM

are you using helm install from the codebase?

Ryan Clark

07/06/2021, 10:16 PM

Yes, Im using this command you gave me earlier:

Copy code

helm install pinot  --values values.yaml -n pinot-quickstart .

Xiang Fu

07/06/2021, 10:16 PM

can you describe your pinot-controller statefulset and paste the output here?

Xiang Fu

07/06/2021, 10:17 PM

and describe the content of pinot-controller inside the configmaps

Fritz Budiyanto

07/06/2021, 10:21 PM

@Ryan Clark I was using codebase release-0.7.1, since top of the tree branch introduced new setting for ZK

urlOverride: "my-zookeeper:2181/my-pinot"

Copy code

git branch
* (HEAD detached at release-0.7.1)
  master

diff --git a/kubernetes/helm/pinot/values.yaml b/kubernetes/helm/pinot/values.yaml
index 5b574aab0..dd170eafe 100644
--- a/kubernetes/helm/pinot/values.yaml
+++ b/kubernetes/helm/pinot/values.yaml
@@ -21,7 +21,7 @@

 image:
   repository: apachepinot/pinot
-  tag: latest
+  tag: release-0.7.1
   pullPolicy: IfNotPresent

 cluster:

Ryan Clark

07/06/2021, 10:56 PM

Copy code

ryanclark@DESKTOP-3TVB8KH pinot % kubectl describe pod/pinot-controller-0 -n pinot-quickstart
Name:         pinot-controller-0
Namespace:    pinot-quickstart
Priority:     0
Node:         ip-192-168-19-85.ec2.internal/192.168.19.85
Start Time:   Tue, 06 Jul 2021 12:14:48 -0500
Labels:       app=pinot
              <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
              <http://app.kubernetes.io/version=0.2.3|app.kubernetes.io/version=0.2.3>
              chart=pinot-0.2.3
              component=controller
              controller-revision-hash=pinot-controller-654bbcbbf7
              <http://helm.sh/chart=pinot-0.2.3|helm.sh/chart=pinot-0.2.3>
              heritage=Helm
              release=pinot
              <http://statefulset.kubernetes.io/pod-name=pinot-controller-0|statefulset.kubernetes.io/pod-name=pinot-controller-0>
Annotations:  <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
Status:       Running
IP:           192.168.3.200
IPs:
  IP:           192.168.3.200
Controlled By:  StatefulSet/pinot-controller
Containers:
  controller:
    Container ID:  <docker://7a1bb1bbcf2f70cbfa1d1ad1314809e8edcd3c7265eeefb29f631d14c142a9a>5
    Image:         apachepinot/pinot:latest-jdk11
    Image ID:      <docker-pullable://apachepinot/pinot@sha256:d56caffcafd469a7c1f4767b73f4d0b4ecc7f3dbbf2bbebb43e1193b28862322>
    Port:          9000/TCP
    Host Port:     0/TCP
    Args:
      StartController
      -configFileName
      /var/pinot/controller/config/pinot-controller.conf
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 06 Jul 2021 17:54:32 -0500
      Finished:     Tue, 06 Jul 2021 17:55:06 -0500
    Ready:          False
    Restart Count:  64
    Environment:
      JAVA_OPTS:  -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log -Dlog4j2.configurationFile=/opt/pinot/conf/log4j2.xml -Dplugins.dir=/opt/pinot/plugins
    Mounts:
      /var/pinot/controller/config from config (rw)
      /var/pinot/controller/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pinot-token-h5krj (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-pinot-controller-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pinot-controller-config
    Optional:  false
  pinot-token-h5krj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pinot-token-h5krj
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  53s (x1392 over 5h39m)  kubelet  Back-off restarting failed container

Xiang Fu

07/06/2021, 11:00 PM

can you check the output of

Copy code

kubectl get configmaps pinot-controller-config -n pinot-quickstart -o yaml

Ryan Clark

07/06/2021, 11:01 PM

Copy code

ryanclark@DESKTOP-3TVB8KH pinot % kubectl describe configmaps pinot-controller-config -n pinot-quickstart
Name:         pinot-controller-config
Namespace:    pinot-quickstart
Labels:       <http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
Annotations:  <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: pinot
              <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: pinot-quickstart

Data
====
pinot-controller.conf:
----
controller.helix.cluster.name=pinot-quickstart
controller.port=9000
controller.data.dir=/var/pinot/controller/data
controller.zk.str=pinot-zookeeper:2181
pinot.set.instance.id.to.hostname=true
controller.task.scheduler.enabled=true
Events:  <none>

Xiang Fu

07/06/2021, 11:01 PM

Xiang Fu

07/06/2021, 11:01 PM

it shows

Copy code

controller.zk.str=pinot-zookeeper:2181

🙌 1

Xiang Fu

07/06/2021, 11:01 PM

hmm

Ryan Clark

07/06/2021, 11:15 PM

Several of my colleagues have the same problem on local K8s (minikube) and aws deployment

Xiang Fu

07/06/2021, 11:16 PM

I think I found the root cause

Xiang Fu

07/06/2021, 11:16 PM

will provide a fix soon

🙌 2

Xiang Fu

07/06/2021, 11:32 PM

https://github.com/apache/incubator-pinot/pull/7131

Xiang Fu

07/07/2021, 9:15 AM

This is fixed

Xiang Fu

07/07/2021, 9:15 AM

can you try to pull with the

latest-jdk11

image?

Ben Gilbert

07/07/2021, 2:17 PM

this fixed the issue I was experiencing, ty @Xiang Fu 🙏

Ryan Clark

07/07/2021, 2:24 PM

@Xiang Fu It works. Thank you for you help!

Open in Slack

Previous Next