👋 hi folks, Question on Zookeeper setup for pinot cluster. We have set up a pinot cluster (2 controllers, 2 brokers, 10 servers) with a single zookeeper instance running on a standalone server with EBS volume deployed using the ZK packaged in the pinot binary. The service was started via

pinot-admin.sh StartZookeeper

. What's the recommended practice for deploying a fault tolerant ZK setup? How to set up the metadata sync across multiple ZK instances?

At least 3 ZK (preferably 5).

Bringing up another ZK instance via

pinot-admin.sh StartZookeeper

did not get the previous metadata from the initial ZK server

Are you on bare metal or using k8s?

not running on k8s, bare ec2

cc: @Xiang Fu

pinot-admin.sh StartZookeeper doesn’t support cluster mode. I would suggest to follow zookeeper doc to start your zk cluster: https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_zkMulitServerSetup

thanks @Xiang Fu, wanted to check if pinot provided a way to run the ZK cluster, but following ZK docs for a separate cluster setup makes sense

yeah, pinot packages zk for usability, but it’s good to set up own zk. Maybe we can package start-zookeeper.sh script directly similar to kafka project

a starter zookeeper script to setup a ZK ensemble for pinot would be great / nice to have

did we add feature to bringup zookeeper cluster using pinot-admin.sh ?

pinot admin script only provide the script to start zk as a single node cluster for poc purposes. For setting up an HA zk cluster, please use the zookeeper binary for that.

Hi @Xiang Fu When Configured External Zookeeper with no SASL configured my pinot was not working. I posted details in thread: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1683558535205369 Now I want to know, does pinot try to connect ZK with SASL by default? If yes do we have any doc where those conf/ default user,pass are provided?

What version of zk are you using? Can you compare it with the version in Pinot helm chart?

we are using 3.4.14 and tried with 3.8.1 but things are not working. what is the recommended version ?