Channels
#troubleshooting
Title
# troubleshooting
a
abhinav wagle
09/22/2022, 5:29 PMHello if we use our zookeeper server, is this the only place we provide the zookeeper url : https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/values.yaml#L521 . Is there any other config/property we need to set for each pinot server, controller, broker so that Pinot pods can talk to zookeeper.
I am seeing this broker log :
Copy code
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/IDEALSTATES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONFIGS/CLUSTER/<redacted>
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONFIGS/PARTICIPANT
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONFIGS/RESOURCE
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/PROPERTYSTORE
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/LIVEINSTANCES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/INSTANCES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/EXTERNALVIEW
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/STATEMODELDEFS
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER/MESSAGES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER/ERRORS
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER/STATUSUPDATES
Invalid cluster setup for cluster: <redacted>, missing znode path: /<redacted>/CONTROLLER/HISTORY
fail to createClient.
org.apache.helix.HelixException: Cluster structure is not set up for cluster: <redacted>
at org.apache.helix.manager.zk.ZKHelixManager.handleNewSession(ZKHelixManager.java:1385) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.manager.zk.ZKHelixManager.createClient(ZKHelixManager.java:782) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.manager.zk.ZKHelixManager.connect(ZKHelixManager.java:819) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.start(BaseBrokerStarter.java:221) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:143) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:278) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:278) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
fail to connect Broker_pinot-broker-0.pinot-broker-headless.<redacted>.svc.cluster.local_8099
org.apache.helix.HelixException: Cluster structure is not set up for cluster: <redacted>
at org.apache.helix.manager.zk.ZKHelixManager.handleNewSession(ZKHelixManager.java:1385) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.manager.zk.ZKHelixManager.createClient(ZKHelixManager.java:782) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.helix.manager.zk.ZKHelixManager.connect(ZKHelixManager.java:819) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.start(BaseBrokerStarter.java:221) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:143) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:278) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:278) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
disconnect Broker_pinot-broker-0.pinot-broker-headless.<redacted>.svc.cluster.local_8099(SPECTATOR) from <redacted>
Shutting down HelixTaskExecutor@Xiang Fu: Any pointers much appreciated. thanks !
I am providing urlOverride: "" in pinot helm file which points to the zookeeper launched via bitnami
x
Xiang Fu
09/22/2022, 8:25 PM
usually your controller should start first to construct the zk folder structure. its requires you pre-create the root directory, e.g.
my-zookeeper:2181/my-pinot/my-pinotOnce controller is up, it will construct helix structure, you can restart broker/server if they happen to start earlier than controller and fail
a
abhinav wagle
09/22/2022, 8:49 PM@Xiang Fu: Ah gotcha. So you are suggesting this :
1. Install zookeeper
2. Create directory on it with folder /my-pinot
3. Then install pinot helm with url override :
my-zookeeper:2181/my-pinotBasically whats happening right now is broker gets started first and all controllers are in
Pendingx
Xiang Fu
09/22/2022, 9:02 PM
hmm
can you describe the controller pod and see why it’s pending ?
a
abhinav wagle
09/22/2022, 9:33 PMOk controller is pending mode cause of lack of cpu on underlying EKS cluster
That makes sense, because of which controller does not start and broker goes into the no directory found state
Thanks @Xiang Fu: Increasing reducing memory/cpu worked. As far as zookeeper is concerned only
urlOverride