Getting this error when trying to set up pinot wit...
# troubleshootingr
Rohit Anilkumar
11/03/2022, 7:45 AMGetting this error when trying to set up pinot with s3 as deep store on an EC2 instance. I am using a zookeeper ensemble and passing the zk addresses as pinot.zk.server=zk1port1,zk2port2….
Executing command: StartController -configFileName conf/pinot-controller.conf
Caught exception while starting controller, exiting.
java.lang.NullPointerException: null
at org.apache.pinot.tools.admin.command.StartControllerCommand.getControllerConf(StartControllerCommand.java:207) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:183) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:165) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:196) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
Rohit Anilkumar
11/03/2022, 7:57 AMpinot.service.role=CONTROLLER
pinot.cluster.name=beaconstac-pinot-dev-cluster
pinot.zk.server=zk1port1,zk2port2,zk3:port3
# Use hostname as Pinot Instance ID other than IP
pinot.set.instance.id.to.hostname=true
# Pinot Controller Port
controller.port=9000
# Pinot Controller VIP Host
controller.vip.host=localhost
# Pinot Controller VIP Port
controller.vip.port=9000
controller.data.dir=s3://bucket/controllerData/
controller.local.temp.dir=/tmp/pinot-tmp-data/
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-east-1
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
Rohit Anilkumar
11/03/2022, 7:57 AMEdited the pinot-controller.conf in /conf directory
m
Mayank
11/03/2022, 2:15 PM
From the stack trace seems it is unable to find the config?
r
Rohit Anilkumar
11/03/2022, 2:20 PMWhich is weird because I have used this command from pinot root directory
bin/pinot-admin.sh StartController -configFileName conf/pinot-controller.conf
Let me give the exact path and try out once again
Rohit Anilkumar
11/03/2022, 2:42 PM@Mayank https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot
Using the exact same command as mentioned here
m
Mayank
11/03/2022, 3:14 PM
Is the dir mounted correctly on the instance
r
Rohit Anilkumar
11/03/2022, 3:15 PMIm on EC2.
m
Mayank
11/03/2022, 3:16 PM
Yes, that’s why I am asking
r
Rohit Anilkumar
11/03/2022, 3:18 PMYea by directory, you mean the conf folder? If yes, its on the instance. I have downloaded the pinot distribution to the instance and edited the pinot-controller.conf under conf/ inside pinot directory
x
Xiang Fu
11/03/2022, 5:02 PM
Have you tried start Pinot with the command zk flag
Xiang Fu
11/03/2022, 5:02 PM
Or in the config file, add controller.zk.str=zk1,zk2
r
Rohit Anilkumar
11/03/2022, 5:21 PMI tried both and getting the same error. Tried -zkAddress as well as through the configuration file. I also tried the default config and the same error is showing up
Rohit Anilkumar
11/03/2022, 5:23 PMOh wait. I used pinot.zk.server in the configuration file. Is there any difference between that and controller.zk.str?
Rohit Anilkumar
11/03/2022, 5:36 PMI have tried all these @Xiang Fu
Copy code
./bin/pinot-admin.sh StartController -zkAddress zk1:2181,zk2:2182,zk3:2183 -configFileName /home/ec2-user/pinot/conf/pinot-controller.conf
./bin/pinot-admin.sh StartController -configFileName /home/ec2-user/pinot/conf/pinot-controller.conf
./bin/pinot-admin.sh StartController -configFileName conf/pinot-controller.conf Copy code
# Pinot Role
pinot.service.role=CONTROLLER
# Pinot Cluster name
pinot.cluster.name=beaconstac-pinot-dev
# Pinot Zookeeper Server
#pinot.zk.server=zk1:2181,zk2:2182,zk3:2183
controller.zk.str=zk1:2181,zk2:2182,zk3:2183
# Use hostname as Pinot Instance ID other than IP
pinot.set.instance.id.to.hostname=true
# Pinot Controller Port
controller.port=9000
# Pinot Controller VIP Host
controller.vip.host=localhost
# Pinot Controller VIP Port
controller.vip.port=9000
# Location to store Pinot Segments pushed from clients
ontroller.data.dir=<s3://path/controllerData/>
controller.local.temp.dir=/tmp/pinot-tmp-data/
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-east-1
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcherRohit Anilkumar
11/03/2022, 5:40 PMI also tried running the default controller config without deep store feature and still getting the same error. Not sure what the issue is.
x
Xiang Fu
11/03/2022, 7:27 PM
hmm, thanks for reporting, I will take a look
Xiang Fu
11/03/2022, 7:28 PM
I think this is not related to s3, but controller zk
Xiang Fu
11/03/2022, 7:28 PM
might related to this one: https://github.com/apache/pinot/issues/9553
r
Rohit Anilkumar
11/03/2022, 7:50 PM@Xiang Fu yea seems like that. But then i have used controller.zk.str in the config and the issue is still there. Does it not accept multiple comma seperated zookeeper IPs?
Rohit Anilkumar
11/03/2022, 7:52 PMRohit Anilkumar
11/03/2022, 11:17 PM@Xiang Fu how about we provide zookeeper addresses as -zkAddress param and remove controller.zk.str from the config? Does startController accept some params from cli and remaining stuff from the configuration file?
x
Xiang Fu
11/03/2022, 11:20 PM
yes, can you try to remove zk conf from the config file and put zk in the command line startController
👍 1
Xiang Fu
11/03/2022, 11:20 PM
Meanwhile I will fix the bug
👍 1
r
Rohit Anilkumar
11/04/2022, 8:40 AM@Xiang Fu seems like that doesnt work.
Copy code
./bin/pinot-admin.sh StartController -zkAddress zk1:2181,zk2:2182,zk3:2183 -configFileName conf/pinot-controller.confRohit Anilkumar
11/05/2022, 1:22 PM@Xiang Fu what would a reasonable timeframe for this to be fixed?
x
Xiang Fu
11/05/2022, 6:03 PM
I saw a fix PR: https://github.com/apache/pinot/pull/9739 We will take a look over weekend
👍 1
Xiang Fu
11/06/2022, 6:23 PM
This is merged, so the new code should fix this problem
Xiang Fu
11/06/2022, 6:23 PM
If you like to build your own binary from current master branchc
👍 2
Xiang Fu
11/06/2022, 6:23 PM
or just use the nightly build tomorrow
6 Views