Hello all, I am encountering an error when trying ...
# generalg
Graham Plata
11/30/2020, 5:07 PMHello all, I am encountering an error when trying to run the sample batch job located here https://docs.pinot.apache.org/basics/getting-started/pushing-your-data-to-pinot in k8s after following this guide https://docs.pinot.apache.org/basics/data-import/pinot-file-system/import-from-gcp with the pinot-gcs plugin. I am assuming I have a config issue somewhere but could use some expertise.
x
Xiang Fu
11/30/2020, 6:09 PM
can you check what’s
JAVA_OPTSg
Graham Plata
11/30/2020, 6:10 PMSure, one moment
Graham Plata
11/30/2020, 6:10 PM Copy code
JAVA_OPTS=-Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/opt/pinot/gc-pinot-controller.log -Dlog4j2.configurationFile=/opt/pinot/conf/pinot-controller-log4j2.xml -Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-gcsx
Xiang Fu
11/30/2020, 6:11 PM
ah
Xiang Fu
11/30/2020, 6:12 PM
can you try remove :`-Dplugins.include=pinot-gcs` or set
-Dplugins.include=pinot-gcs,pinot-csv,pinot-batch-ingestion-standaloneg
Graham Plata
11/30/2020, 6:12 PMCan do one moment
x
Xiang Fu
11/30/2020, 6:12 PM
Copy code
JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins" bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile job.yamlXiang Fu
11/30/2020, 6:13 PM
you can just run this in the container
g
Graham Plata
11/30/2020, 6:21 PM Copy code
root@pinot-controller-0:/opt/pinot# JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins" bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile /var/pinot/controller/data/job.yml
2020/11/30 18:21:31.169 ERROR [LaunchDataIngestionJobCommand] [main] Got exception to kick off standalone data ingestion job -
java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:144) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:113) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:123) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
Caused by: java.lang.NullPointerException
at shaded.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:770) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at com.google.cloud.storage.BucketInfo$BuilderImpl.build(BucketInfo.java:1313) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at com.google.cloud.storage.BucketInfo.of(BucketInfo.java:1755) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:209) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.getBucket(GcsPinotFS.java:87) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.isDirectory(GcsPinotFS.java:358) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.run(SegmentGenerationJobRunner.java:154) ~[pinot-batch-ingestion-standalone-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:142) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
... 4 more
2020/11/30 18:21:31.177 ERROR [PinotAdministrator] [main] Exception caught:
java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:144) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:113) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:123) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
Caused by: java.lang.NullPointerException
at shaded.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:770) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at com.google.cloud.storage.BucketInfo$BuilderImpl.build(BucketInfo.java:1313) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at com.google.cloud.storage.BucketInfo.of(BucketInfo.java:1755) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:209) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.getBucket(GcsPinotFS.java:87) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.plugin.filesystem.GcsPinotFS.isDirectory(GcsPinotFS.java:358) ~[pinot-gcs-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.run(SegmentGenerationJobRunner.java:154) ~[pinot-batch-ingestion-standalone-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:142) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-318c1077bb4a8aa74a03acad8f236aff8eb9fa0d]
... 4 morex
Xiang Fu
11/30/2020, 6:23 PM
I think you need to specify the bucket in the input/output dir
Xiang Fu
11/30/2020, 6:23 PM
same for controller config
g
Graham Plata
11/30/2020, 6:24 PMYeah, I was doing some tinkering earlier while I was troubleshooting. Let me clean that up before re-running
x
Xiang Fu
11/30/2020, 7:03 PM
sure