https://pinot.apache.org/ logo
Join Slack
Powered by
Hi Team, I'm facing issue while loading data from ...
# troubleshooting
s
Hi Team, I'm facing issue while loading data from s3 to pinot offline table. I'm using Ec2 for pinot setup. I'm able to access my s3 files from ec2 instance but data is not loading into pinot table. To this I also setup my minion job and enabled scheduler in controller config with 
controller.task.scheduler.enabled=true
No luck here, when I check the scheduler information with API it returns this
k
@Sowmya Gowda Didn't changing the config work as we discussed last time, the changes you were making to the config were getting lost during restart since it was not mounted outside docker
s
I tried it with mounting and facing the same issue.
k
Can you share the docker command
s
docker command of what ?
k
launching controller/pinot docker container after making changes
s
docker-compose up
k
In that case, share the yaml file
s
docker-compose.yml
docker-compose.yml
k
can you share the one where you have mounted the conf directory?
s
using volumes only right which I have mentioned in the yml file
k
Copy code
version: '3.8'
services:
  zookeeper:
    image: zookeeper:3.5.6
    hostname: zookeeper
    container_name: manual-zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - ./pinot-demo_default/zookeeper/data:/data
      - ./pinot-demo_default/zookeeper/datalog:/datalog
  pinot-controller:
    image: apachepinot/pinot:latest
    command: "StartController -zkAddress manual-zookeeper:2181 -configFileName /opt/pinot/conf/pinot-controller.conf"
    container_name: "manual-pinot-controller"
    restart: unless-stopped
    ports:
      - "9000:9000"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log"
    depends_on:
      - zookeeper
    volumes:
      - ./pinot-demo_default/pinot/controller:/tmp/pinot/data/controller
      - ./pinot-demo-default/pinot/controller-conf/:/opt/pinot/conf
  pinot-broker:
    image: apachepinot/pinot:latest
    command: "StartBroker -zkAddress manual-zookeeper:2181"
    restart: unless-stopped
    container_name: "manual-pinot-broker"
    ports:
      - "8099:8099"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log"
    depends_on:
      - pinot-controller
  pinot-server:
    image: apachepinot/pinot:latest
    command: "StartServer -zkAddress manual-zookeeper:2181"
    restart: unless-stopped
    container_name: "manual-pinot-server"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log"
    depends_on:
      - pinot-broker
    volumes:
      - ./pinot-demo_default/pinot/server:/tmp/data/server
  pinot-minion:
    image: apachepinot/pinot:latest
    command: "StartMinion -zkAddress manual-zookeeper:2181"
    restart: unless-stopped
    container_name: "manual-pinot-minion" 
    volumes:
      - ./pinot-demo_default/pinot/minion:/tmp/data/minion
volumes:
  pinot-demo_default: {}
Try this. Add the scheduler enable config in 
./pinot-demo-default/pinot/controller-conf/pinot-controller.conf
file and restart docker
s
SUre
There is an issue with the path
Copy code
manual-pinot-controller | 2022/06/27 14:53:26.582 INFO [StartControllerCommand] [main] Executing command: StartController -configFileName /opt/pinot/conf/pinot-controller.conf
manual-pinot-controller | 2022/06/27 14:53:26.582 ERROR [StartControllerCommand] [main] Caught exception while starting controller, exiting.
manual-pinot-controller | java.lang.RuntimeException: Error: Unable to find controller config file /opt/pinot/conf/pinot-controller.conf
manual-pinot-controller | 	at org.apache.pinot.tools.utils.PinotConfigUtils.generateControllerConf(PinotConfigUtils.java:85) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.command.StartControllerCommand.getControllerConf(StartControllerCommand.java:203) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:183) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:165) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller | 	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:196) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-6582750ea6419233eb1fe5a6d196921d23ee89c1]
manual-pinot-controller exited with code 255
k
Can you do a ls inside controller and check if the directory '/opt/pinot` exists and if it has a conf dir?
s
Yes conf path is there and same
Open in Slack
PreviousNext
Powered by