Hey I m working on POC for Pinot for our team but I have a p

Question

Hey I m working on POC for Pinot for our team but I have a problem deploying it to AWS I set up the k8n cluster on EKS by following the instructions then I moved to start Pinot with Helm but the broker keeps crashing everything else is in the pending state logs for broker show `NullPointerException` more context on screenshots I cannot run locally as well I have M1 macbook I tried to compile it from the source but it failed with `InvocationTargetException`

Wojciech Wasik · Answer

Any help appreciated

Kishore G · Answer

please check the resources in values in helm chart

Wojciech Wasik · Answer

<https github com apache pinot blob master kubernetes helm pinot values yaml> I have not changed anything in this file

Mayank · Answer

eyes

Xiaobing · Answer

it s interesting that broker tried to get started before ZK was fully up as in the first screenshot and broker error seems like related to connecting with ZK

Mayank · Answer

Right

Xiaobing · Answer

I ve noticed two recent changes on those helm chart files 1 bumped ZK from 7 to 9 2 added explicit resource requirements so I d wonder 1 how much resource provided for your pods e g mem is required to be 1 25GB in the helm values yaml file 2 any error logs from that Zookeeper pod

Xiang Fu · Answer

first time requires controller to go up

Xiang Fu · Answer

you can give more resources to your k8s or just update helm to start with less resources

Wojciech Wasik · Answer

< Xiaobing> 1 not sure I have just a basic understanding about k8n and helm but k8n cluster is on the t3 xlarge 2 no logs

Xiaobing · Answer

not sure if EKS has cmd like `kubectl describe pod` to describe what s happening for the pods also perhaps give it a try to remove those new resource requirements added in values yaml by this PR <https github com apache pinot pull 9012 files> and see how it goes

Xiaobing · Answer

gt first time requires controller to go up cc < Xiang Fu> is there a way to enforce the starting order

Wojciech Wasik · Answer

I tried 1 set less resources in values yml 2 remove resource requirements it failed for both but the second change gives a different exception although still about not able to connect to ZK

Xiaobing · Answer

got it at this point I d focus on to getting ZK then Controller up firstly

Xiang Fu · Answer

you need resources to start all the pods

Xiang Fu · Answer

can you run `kubectl describe nodes`

Xiang Fu · Answer

< Wojciech Wasik>

Xiang Fu · Answer

you can also change wherever ```resources requests memory 1 25Gi ``` to ``` resources ``` in values yaml

Xiang Fu · Answer

```helm install pinot pinot pinot n pinot quickstart values values yaml```

Xiang Fu · Answer

In case you can run below to get the `values yaml` file ```helm inspect values pinot pinot gt values yaml```

Wojciech Wasik · Answer

Thank you < Xiang Fu> that s all very useful

Wojciech Wasik · Answer

I checked the `kubectl describe nodes` with default `values yaml` file

Wojciech Wasik · Answer

```Name ip 192 168 15 222 ec2 internal Roles lt none gt Labels Annotations 192 168 15 222 0 true CreationTimestamp Wed 28 Sep 2022 11 54 35 +0200 Taints lt none gt Unschedulable false Lease HolderIdentity ip 192 168 15 222 ec2 internal AcquireTime lt unset gt RenewTime Thu 29 Sep 2022 09 16 30 +0200 Conditions Type Status LastHeartbeatTime LastTransitionTime Reason Message MemoryPressure False Thu 29 Sep 2022 09 12 15 +0200 Wed 28 Sep 2022 11 54 33 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu 29 Sep 2022 09 12 15 +0200 Wed 28 Sep 2022 11 54 33 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu 29 Sep 2022 09 12 15 +0200 Wed 28 Sep 2022 11 54 33 +0200 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu 29 Sep 2022 09 12 15 +0200 Wed 28 Sep 2022 11 56 48 +0200 KubeletReady kubelet is posting ready status Addresses InternalIP 192 168 15 222 ExternalIP 54 224 213 7 Hostname ip 192 168 15 222 ec2 internal InternalDNS ip 192 168 15 222 ec2 internal ExternalDNS Capacity attachable volumes aws ebs 25 cpu 4 ephemeral storage 83873772Ki hugepages 1Gi 0 hugepages 2Mi 0 memory 16203828Ki pods 58 Allocatable attachable volumes aws ebs 25 cpu 3920m ephemeral storage 76224326324 hugepages 1Gi 0 hugepages 2Mi 0 memory 15186996Ki pods 58 System Info Machine ID ec2485bc260c53cc76d5e3608c3e4e86 System UUID ec2485bc 260c 53cc 76d5 e3608c3e4e86 Boot ID 91abe21a fa0a 4701 b593 d129d2afc631 Kernel Version 5 4 209 116 367 amzn2 x86 64 OS Image Amazon Linux 2 Operating System linux Architecture amd64 Container Runtime Version Kubelet Version v1 23 9 eks ba74326 Kube Proxy Version v1 23 9 eks ba74326 ProviderID aws us east 1c i 08c3cebafc6f15b28 Non terminated Pods 5 in total Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age kube system aws node qm7m7 25m 0 0 0 0 0 0 0 21h kube system coredns d5b9bfc4 qvb7v 100m 2 0 0 70Mi 0 170Mi 1 21h kube system coredns d5b9bfc4 t5kqp 100m 2 0 0 70Mi 0 170Mi 1 21h kube system kube proxy lgv9w 100m 2 0 0 0 0 0 0 21h pinot quickstart pinot broker 0 0 0 0 0 0 0 0 0 99s Allocated resources Total limits may be over 100 percent i e overcommitted Resource Requests Limits cpu 325m 8 0 0 memory 140Mi 0 340Mi 2 ephemeral storage 0 0 0 0 hugepages 1Gi 0 0 0 0 hugepages 2Mi 0 0 0 0 attachable volumes aws ebs 0 0 Events lt none gt ```

Wojciech Wasik · Answer

It does not seem to be a resources memory problem but I ll recheck the changing resources to `resources `

Xiang Fu · Answer

yeah your node is 4cpu 16g ram so just change resources to Another thing I want to check is what s your storageclass type can you run ```kubectl get storageclass```

Xiang Fu · Answer

double check if you can use ebs as default storage class

Xiang Fu · Answer

the storage class name should be gp2

Wojciech Wasik · Answer

right now it might be not `gp2` because i reverted my initial changes but I think I tried that too let me double check

Wojciech Wasik · Answer

`kubectl get storageclass` gives `gp2` but shiould I change the ``` persistence storageClass ```

Wojciech Wasik · Answer

to gp2

Wojciech Wasik · Answer

in values yaml

Xiang Fu · Answer

yes

Xiang Fu · Answer

So zk is starting right

Xiang Fu · Answer

hmmm

Xiang Fu · Answer

can you do `kubectl describe pod pinot zookeeper 0`

Wojciech Wasik · Answer

````Name pinot zookeeper 0 Namespace pinot quickstart Priority 0 Node lt none gt Labels controller revision hash=pinot zookeeper 546f7fdcdb Annotations eks privileged Status Pending IP IPs lt none gt Controlled By StatefulSet pinot zookeeper Containers zookeeper Image Ports 2181 TCP 2888 TCP 3888 TCP Host Ports 0 TCP 0 TCP 0 TCP Command scripts setup sh Requests cpu 250m memory 256Mi Liveness exec bin bash c echo ruok | timeout 2 nc w 2 localhost 2181 | grep imok delay=30s timeout=5s period=10s success=1 failure=6 Readiness exec bin bash c echo ruok | timeout 2 nc w 2 localhost 2181 | grep imok delay=5s timeout=5s period=10s success=1 failure=6 Environment BITNAMI DEBUG false ZOO DATA LOG DIR ZOO PORT NUMBER 2181 ZOO TICK TIME 2000 ZOO INIT LIMIT 10 ZOO SYNC LIMIT 5 ZOO PRE ALLOC SIZE 65536 ZOO SNAPCOUNT 100000 ZOO MAX CLIENT CNXNS 60 ZOO 4LW COMMANDS WHITELIST srvr mntr ruok ZOO LISTEN ALLIPS ENABLED no ZOO AUTOPURGE INTERVAL 1 ZOO AUTOPURGE RETAIN COUNT 5 ZOO MAX SESSION TIMEOUT 40000 ZOO SERVERS pinot zookeeper 0 pinot zookeeper headless pinot quickstart svc cluster local 2888 3888 1 ZOO ENABLE AUTH no ZOO HEAP SIZE 1024 ZOO LOG LEVEL ERROR ALLOW ANONYMOUS LOGIN yes POD NAME pinot zookeeper 0 v1 metadata name Mounts bitnami zookeeper from data rw scripts setup sh from scripts rw path= setup sh var run secrets kubernetes io serviceaccount from kube api access c9mj5 ro Conditions Type Status PodScheduled False Volumes data Type PersistentVolumeClaim a reference to a PersistentVolumeClaim in the same namespace ClaimName data pinot zookeeper 0 ReadOnly false scripts Type ConfigMap a volume populated by a ConfigMap Name pinot zookeeper scripts Optional false kube api access c9mj5 Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class Burstable Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events Type Reason Age From Message Warning FailedScheduling 3m48s x3 over 23m default scheduler running PreBind plugin VolumeBinding binding volumes timed out waiting for the condition```

Xiang Fu · Answer

hmm did you set ```storageClass gp2 ```

Xiang Fu · Answer

can you keep this file to `gp2 yaml` ```apiVersion kind StorageClass metadata name gp2 annotations true parameters fsType ext4 type gp2 provisioner reclaimPolicy Delete volumeBindingMode WaitForFirstConsumer``` Then run `kubectl apply f gp2 yaml`

Xiang Fu · Answer

it should create you a ebs disk

Xiang Fu · Answer

something interesting is that your k8s has no default storage class

Wojciech Wasik · Answer

There s no more warning about volume ``` kubectl describe pod pinot zookeeper 0 Name pinot zookeeper 0 Namespace pinot quickstart Priority 0 Node lt none gt Labels controller revision hash=pinot zookeeper 64bf956ff8 Annotations eks privileged Status Pending IP IPs lt none gt Controlled By StatefulSet pinot zookeeper Containers zookeeper Image Ports 2181 TCP 2888 TCP 3888 TCP Host Ports 0 TCP 0 TCP 0 TCP Command bash ec Execute entrypoint as usual after obtaining ZOO SERVER ID check ZOO SERVER ID in persistent volume via myid if not present set based on POD hostname if f bitnami zookeeper data myid then export ZOO SERVER ID= $ cat bitnami zookeeper data myid else HOSTNAME=`hostname s` if $HOSTNAME =~ 0 9 + $ then ORD=$ BASH REMATCH 2 export ZOO SERVER ID=$ ORD + 1 else echo Failed to get index from hostname $HOST exit 1 fi fi exec entrypoint sh run sh Requests cpu 250m memory 256Mi Liveness exec bin bash c echo ruok | timeout 2 nc w 2 localhost 2181 | grep imok delay=30s timeout=5s period=10s success=1 failure=6 Readiness exec bin bash c echo ruok | timeout 2 nc w 2 localhost 2181 | grep imok delay=5s timeout=5s period=10s success=1 failure=6 Environment ZOO DATA LOG DIR ZOO PORT NUMBER 2181 ZOO TICK TIME 2000 ZOO INIT LIMIT 10 ZOO SYNC LIMIT 5 ZOO MAX CLIENT CNXNS 60 ZOO 4LW COMMANDS WHITELIST srvr mntr ruok ZOO LISTEN ALLIPS ENABLED no ZOO AUTOPURGE INTERVAL 0 ZOO AUTOPURGE RETAIN COUNT 3 ZOO MAX SESSION TIMEOUT 40000 ZOO SERVERS pinot zookeeper 0 pinot zookeeper headless pinot quickstart svc cluster local 2888 3888 1 ZOO ENABLE AUTH no ZOO HEAP SIZE 1024 ZOO LOG LEVEL ERROR ALLOW ANONYMOUS LOGIN yes POD NAME pinot zookeeper 0 v1 metadata name Mounts bitnami zookeeper from data rw var run secrets kubernetes io serviceaccount from kube api access fr8kw ro Volumes data Type PersistentVolumeClaim a reference to a PersistentVolumeClaim in the same namespace ClaimName data pinot zookeeper 0 ReadOnly false kube api access fr8kw Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class Burstable Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events lt none gt ```

Wojciech Wasik · Answer

but the `broker` still crashing because of fail to connect to ZK and ZK is in pending state

Wojciech Wasik · Answer

and I do ```helm install pinot pinot pinot n pinot quickstart values values yaml```

Wojciech Wasik · Answer

Ok I think I solved the issue the problem was that k8n cluster by default did not have the required volume controller

Xiang Fu · Answer

glad you solved it Is this a AWS EKS cluster Technically all those things are built in

Mayank · Answer

< Xiaobing> let s document the learnings in the docs page so it helps the next person

Wojciech Wasik · Answer

Yes it s AWS EKS I used exactly this command ```EKS CLUSTER NAME=pinot quickstart eksctl create cluster name $ EKS CLUSTER NAME version 1 16 region us west 2 nodegroup name standard workers node type t3 xlarge nodes 1 nodes min 1 nodes max 1```

Wojciech Wasik · Answer

actually one change I updated the version to `1 23`

Wojciech Wasik · Answer

1 16 is not available anymore

Xiang Fu · Answer

got it we will update the script as well and see if the volume controller issue we can solve

reallyonthemove tous · Answer

looks like i am hitting the same issue binding volumes timed out waiting for the condition I have edited the values yaml file to change the storageClass gp2 and set resources k8 cluster created with eksctl create cluster name $ EKS CLUSTER NAME version 1 23 region us east 1 nodegroup name standard workers node type t3 xlarge nodes 1 nodes min 1 nodes max 1 pinot cluster installed with helm install pinot pinot pinot n pinot quickstart values values yaml < Xiang Fu> what updates do we need to make to the script