Hello if we use our zookeeper server, is this the ...
# troubleshootinga
abhinav wagle
09/22/2022, 5:29 PMHello if we use our zookeeper server, is this the only place we provide the zookeeper url : https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/values.yaml#L521 . Is there any other config/property we need to set for each pinot server, controller, broker so that Pinot pods can talk to zookeeper.
abhinav wagle
09/22/2022, 6:31 PMI am seeing this broker log :
Copy code
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/IDEALSTATES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONFIGS/CLUSTER/<redacted>
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONFIGS/PARTICIPANT
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONFIGS/RESOURCE
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/PROPERTYSTORE
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/LIVEINSTANCES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/INSTANCES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/EXTERNALVIEW
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/STATEMODELDEFS
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER/MESSAGES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER/ERRORS
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER/STATUSUPDATES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER/HISTORY
fail to createClient.
org.apache.helix.HelixException: Cluster structure is not set up for cluster: <redacted>
at org.apache.helix.manager.zk.ZKHelixManager.handleNewSession(ZKHelixManager.java:1385) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.manager.zk.ZKHelixManager.createClient(ZKHelixManager.java:782) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.manager.zk.ZKHelixManager.connect(ZKHelixManager.java:819) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.start(BaseBrokerStarter.java:221) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:143) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:278) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:278) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
fail to connect Broker_pinot-broker-0.pinot-broker-headless.<redacted>.svc.cluster.local_8099
org.apache.helix.HelixException: Cluster structure is not set up for cluster: <redacted>
at org.apache.helix.manager.zk.ZKHelixManager.handleNewSession(ZKHelixManager.java:1385) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.manager.zk.ZKHelixManager.createClient(ZKHelixManager.java:782) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.manager.zk.ZKHelixManager.connect(ZKHelixManager.java:819) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.start(BaseBrokerStarter.java:221) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:143) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:278) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:278) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
disconnect Broker_pinot-broker-0.pinot-broker-headless.<redacted>.svc.cluster.local_8099(SPECTATOR) from <redacted>
Shutting down HelixTaskExecutorabhinav wagle
09/22/2022, 7:47 PM@Xiang Fu: Any pointers much appreciated. thanks !
abhinav wagle
09/22/2022, 7:48 PMI am providing urlOverride: "" in pinot helm file which points to the zookeeper launched via bitnami
x
Xiang Fu
09/22/2022, 8:25 PM
usually your controller should start first to construct the zk folder structure. its requires you pre-create the root directory, e.g.
my-zookeeper:2181/my-pinot/my-pinotXiang Fu
09/22/2022, 8:26 PM
Once controller is up, it will construct helix structure, you can restart broker/server if they happen to start earlier than controller and fail
a
abhinav wagle
09/22/2022, 8:49 PM@Xiang Fu: Ah gotcha. So you are suggesting this :
abhinav wagle
09/22/2022, 8:50 PM1. Install zookeeper
2. Create directory on it with folder /my-pinot
3. Then install pinot helm with url override :
my-zookeeper:2181/my-pinotabhinav wagle
09/22/2022, 8:57 PMBasically whats happening right now is broker gets started first and all controllers are in
Pendingx
Xiang Fu
09/22/2022, 9:02 PM
hmm
Xiang Fu
09/22/2022, 9:02 PM
can you describe the controller pod and see why it’s pending ?
a
abhinav wagle
09/22/2022, 9:33 PMOk controller is pending mode cause of lack of cpu on underlying EKS cluster
abhinav wagle
09/22/2022, 9:34 PMThat makes sense, because of which controller does not start and broker goes into the no directory found state
abhinav wagle
09/22/2022, 10:07 PMThanks @Xiang Fu: Increasing reducing memory/cpu worked. As far as zookeeper is concerned only
urlOverride👍 1