https://pinot.apache.org/ logo
Join Slack
Powered by
Hi folks! I'd like to ask for some help on running...
# troubleshooting
d
Hi folks! I'd like to ask for some help on running Pinot locally on a k3s cluster (tiny k8s cluster for development). I'm migrating my Pinot instances manifests from 
Deployment
to 
StatefulSet
and facing some issues. More in this thread.
The reason why I'm doing this move is so that I can, together with some configurations, have more persistent names and in the end avoid losing integration between computer restarts, which is causing my local segments to not be found anymore.
This is what I'm having with the Broker for example:
Copy code
$ kubectl logs pinot-broker-0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/pinot/lib/pinot-all-0.10.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-metrics/pinot-dropwizard/pinot-dropwizard-0.10.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-metrics/pinot-yammer/pinot-yammer-0.10.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-input-format/pinot-parquet/pinot-parquet-0.10.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-environment/pinot-azure/pinot-azure-0.10.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.10.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <http://www.slf4j.org/codes.html#multiple_bindings> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-0.10.0-jar-with-dependencies.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022/07/07 01:18:34.407 INFO [StartBrokerCommand] [main] Executing command: StartBroker -zkAddress zookeeper:2181 -configFileName /tmp/config/pinot-broker.conf
2022/07/07 01:18:34.454 INFO [StartServiceManagerCommand] [main] Executing command: StartServiceManager -clusterName pinot -zkAddress zookeeper:2181 -port -1 -bootstrapServices []
2022/07/07 01:18:34.454 INFO [StartServiceManagerCommand] [main] Starting a Pinot [SERVICE_MANAGER] at 0.444s since launch
2022/07/07 01:18:34.458 INFO [StartServiceManagerCommand] [main] Started Pinot [SERVICE_MANAGER] instance [ServiceManager_pinot-broker-0.pinot-broker.brandindex.svc.cluster.local_-1] at 0.448s since launch
2022/07/07 01:18:34.461 INFO [StartServiceManagerCommand] [Start a Pinot [BROKER]] Starting a Pinot [BROKER] at 0.45s since launch
2022/07/07 01:18:38.801 ERROR [ZKHelixManager] [Start a Pinot [BROKER]] fail to createClient.
org.apache.helix.HelixException: Cluster structure is not set up for cluster: pinot
	at org.apache.helix.manager.zk.ZKHelixManager.handleNewSession(ZKHelixManager.java:1124) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
	at org.apache.helix.manager.zk.ZKHelixManager.createClient(ZKHelixManager.java:701) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
	at org.apache.helix.manager.zk.ZKHelixManager.connect(ZKHelixManager.java:738) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
	at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.start(BaseBrokerStarter.java:209) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
	at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:143) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:276) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:302) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:276) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
2022/07/07 01:18:38.812 ERROR [ZKHelixManager] [Start a Pinot [BROKER]] fail to connect Broker_pinot-broker-0.pinot-broker.brandindex.svc.cluster.local_8099
The message 
Cluster structure is not set up for cluster: pinot
is not clear to me; It does refer to the cluster name I used, but I'm not sure how to fix it.
m
Are you starting ZK and Controller before Broker
d
Ah, nevermind! Found the issue! Not really, because when deploying with straight k8s manifests (which is what I need to do locally) I can't have a start order (or at least I haven't found a way to do that yet). But the problem was that the Controller had an error in the volumes configurations, so the Broker was unable to find it. Now it's running fine 🙂
Hopefully now I'll be able to work on the segments without losing them between computer restarts... hahahaha
Success!!! Restarted my computer and still got all my segments there! Whew...
👍 1
Open in Slack
PreviousNext
Powered by