https://pinot.apache.org/ logo
#general
Title
# general
r

RK

05/06/2021, 11:52 AM
Hello everyone I have recently started Using Apache Pinot and I am able to integrate my Kafka with Pinot. I have done the setup in local and once I started  Pinot using bin/quick-start-batch.sh. I am able to see all the Pinot details on localhost:9000. I want to add one user Authentication feature here so someone use this localhost:9000. It should ask for credential and then it should go to pinot home page. I checked multiple documents and youtube videos but could not find any reference for the same. Kindly suggest /guide me how Can i implement the same.
m

Mayank

05/06/2021, 1:07 PM
@User
r

RK

05/06/2021, 3:56 PM
Team Someone Please help.
m

Mayank

05/06/2021, 4:01 PM
I know that @User has added support for authentication. Let's give him a minute. In the meanwhile I can try to find the PRs he added.
r

RK

05/06/2021, 4:49 PM
Okay thanks @User
m

Mayank

05/06/2021, 5:03 PM
You can refer to this PR for now: https://github.com/apache/incubator-pinot/pull/6613 @User
a

Alexander Pucher

05/06/2021, 5:38 PM
Hi RK - the PR above actually achieves the scenario you describe. It’ll officially become available in Pinot 0.8.0+, but you can already use the snapshot to try out the feature. There’s already a configuration reference here: https://docs.pinot.apache.org/operators/tutorials/authentication-and-authorization
r

RK

05/07/2021, 2:34 AM
@User inside bin folder I have multiple .Sh files kindly guide me in which file I need to add all these properties. . And after adding all these properties simply I need to start bin/quick-start-batch.sh or need to do some additional setups. controller.admin.access.control.principals=admin controller.admin.access.control.principals.admin.password=verysecret controller.admin.access.control.factory.class=org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory controller.admin.access.control.principals=admin,user controller.admin.access.control.principals.admin.password=verysecret controller.admin.access.control.principals.user.password=secret controller.admin.access.control.principals.user.tables=myusertable,baseballStats,stuff controller.admin.access.control.principals.user.permissions=READ # the factory class property is different for the broker pinot.broker.access.control.class=org.apache.pinot.broker.broker.BasicAuthAccessControlFactory pinot.broker.access.control.principals=admin,user pinot.broker.access.control.principals.admin.password=verysecret pinot.broker.access.control.principals.user.password=secret pinot.broker.access.control.principals.user.tables=baseballStats,otherstuff
@User
a

Alexander Pucher

05/07/2021, 7:09 PM
Pinot is typically configured using *.properties files. You can create e.g.
controller.properties
and
broker.properties
and store the respective properties there. Then, you can use the various starter scripts like
bin/start-controller.sh -configFileName controller.properties
to bring up the individual pinot components with you new configs. (The commands also support a
--help
flag, and there’s a detailed reference here: https://docs.pinot.apache.org/basics/getting-started/advanced-pinot-setup)
I’ve also added a preconfigured
quick-start-auth.sh
all-in-one script, but it may take a day or two to filter through the review and snapshot process (PR here: https://github.com/apache/incubator-pinot/pull/6892)
r

RK

05/08/2021, 4:18 AM
1. Started Zookeer Manually using below command: bin/pinot-admin.sh StartZookeeper \ -zkPort 2191 -----------------------_------ 2. Created One controller.properties file in bin folder and added following properties. controller.admin.access.control.principals=admin controller.admin.access.control.principals.admin.password=verysecret
 controller.segment.fetcher.auth.token=Basic YWRtaW46dmVyeXNlY3JldA controller.admin.access.control.factory.class=org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory controller.admin.access.control.principals=admin,user 
 controller.admin.access.control.principals.admin.password=verysecret
 controller.admin.access.control.principals.user.password=secret
 controller.admin.access.control.principals.user.tables=myusertable,baseballStats
 ,stuff controller.admin.access.control.principals.user.permissions=READ ------------------------------------------------ 3. Added these 3 extra properties also since it was throwing error to add these parameter in config file. controller.access.protocols.http.port=2191 controller.zk.str=localhost:2181 controller.helix.cluster.name=PinotCluster i.e. : 2021/05/08 092655.054 ERROR [StartControllerCommand] [main] Caught exception while starting controller, exiting. org.apache.commons.configuration.ConfigurationException: Pinot Controller Config Validation Error: missing controller http port, please fix 'controller.access.protocols.http.port' property in config file. ------------------------------------------------ 4. After then when I am trying to start controller with configFileName property its giving below error. apache-pinot-incubating-0.7.1-bin % bin/start-controller.sh \ -configFileName bin/controller.properties 2021/05/08 092943.096 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 0.051 since launch java.lang.IndexOutOfBoundsException: Index: 0, Size: 0     at java.util.ArrayList.rangeCheck(ArrayList.java:659) ~[?:1.8.0_261]     at java.util.ArrayList.get(ArrayList.java:435) ~[?:1.8.0_261]     at org.apache.pinot.controller.ControllerStarter.<init>(ControllerStarter.java:156) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:115) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:91) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:234) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:286) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:233) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:183) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:130) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     at org.apache.pinot.tools.admin.PinotController.main(PinotController.java:35) [pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]
@User kindly suggest.
a

Alexander Pucher

05/08/2021, 6:42 AM
Hi RK - in a few hours the
quick-start-auth.sh
should become available if you don’t want to wait try this - looks like the ports got swapped: (1) instead of
controller.access.protocols.http.port=2191
simply use
controller.port=9000
and
controller.host=localhost
(2) instead of
controller.zk.str=localhost:2181
go with
controller.zk.str=localhost:2191
(or whichever port you start zookeeper on)
(3) you may also need to set the datadir e.g. like so
controller.data.dir=/tmp/mydatadir
if you’re pulling from github master directly, you can already generate the quickstart script with a bindist build
r

RK

05/08/2021, 3:15 PM
When I am executing bin/quick-start-batch.sh. In log I found Executing command: StartZookeeper -zkPort 2123 -dataDir /var/folders/5f/c00hxf596s19gdw84svdm7j40000gp/T/1620485847863/baseballStats/rawdata/PinotZkDir Start zookeeper at localhost:2123 in thread main Executing command: StartController -clusterName QuickStartCluster -controllerHost null -controllerPort 9000 -dataDir /var/folders/5f/c00hxf596s19gdw84svdm7j40000gp/T/1620485847863/baseballStats/rawdata/PinotControllerDir0 -zkAddress localhost:2123 Executing command: StartServiceManager -clusterName QuickStartCluster -zkAddress. so In my controller.properties I have mentioned same details for controller.zk.str=localhost:2123 controller.data.dir=/var/folders/5f/c00hxf596s19gdw84svdm7j40000gp/T/1620485357791/baseballStats/rawdata/PinotZkDir controller.helix.cluster.name=QuickStartCluster PFB the controller.properties file. controller.admin.access.control.principals=admin controller.admin.access.control.principals.admin.password=verysecret controller.segment.fetcher.auth.token=Basic YWRtaW46dmVyeXNlY3JldA controller.admin.access.control.factory.class=org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory controller.admin.access.control.principals=admin,user controller.admin.access.control.principals.admin.password=verysecret controller.admin.access.control.principals.user.password=secret controller.admin.access.control.principals.user.tables=myusertable,baseballStats
 ,stuff controller.admin.access.control.principals.user.permissions=READ controller.port=9000 ontroller.host=localhost controller.zk.str=localhost:2123 controller.data.dir=/var/folders/5f/c00hxf596s19gdw84svdm7j40000gp/T/1620485357791/baseballStats/rawdata/PinotZkDir controller.helix.cluster.name=QuickStartCluster After That when I am starting controller with controller.properties File. I am getting java.lang.NullPointerException. Kindly Help. PFA the Log File for Controller Error.
@User kindly suggest.
a

Alexander Pucher

05/08/2021, 8:48 PM
Kindly pull and build the most recent
master
of incubator-pinot. You’ll now find a
quick-start-auth.sh
script which launches a pre-configured pinot cluster with auth