i m trying to run a batch ingest job using aws emr i m runni Apache Pinot #troubleshooting

i’m trying to run a batch ingest job using aws emr...

Nicholas Yu

12/16/2021, 2:17 AM

i’m trying to run a batch ingest job using aws emr. i’m running a spark submit step like so

Copy code

spark-submit 
--master yarn
--class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand 
s3://<bucket_name>/lib/pinot-all-0.9.0-jar-with-dependencies.jar -jobSpecFile s3://<bucket_name>/jobs/<table_name>/job.yaml

but i’m getting a java.lang.NoSuchMethodException: org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main

Xiang Fu

12/16/2021, 5:45 AM

I think this command is recently removed, try this one:

Copy code

--class org.apache.pinot.tools.admin.PinotAdministrator
s3://<bucket_name>/lib/pinot-all-0.9.0-jar-with-dependencies.jar LaunchDataIngestionJob -jobSpecFile s3://<bucket_name>/jobs/<table_name>/job.yaml

Chris Theodore Jayakumar

12/17/2021, 1:56 AM

we tried this… and we are getting an exception at

Copy code

Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.apache.pinot.tools.admin.command.StartKafkaCommand.<init>(StartKafkaCommand.java:51)
	at org.apache.pinot.tools.admin.PinotAdministrator.<clinit>(PinotAdministrator.java:98)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:207)
	at <http://org.apache.spark.deploy.SparkSubmit.org|org.apache.spark.deploy.SparkSubmit.org>$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1047)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1056)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.NoSuchElementException
	at java.base/java.util.ServiceLoader$2.next(ServiceLoader.java:1309)
	at java.base/java.util.ServiceLoader$2.next(ServiceLoader.java:1297)
	at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1395)
	at org.apache.pinot.tools.utils.KafkaStarterUtils.getKafkaConnectorPackageName(KafkaStarterUtils.java:54)
	at org.apache.pinot.tools.utils.KafkaStarterUtils.<clinit>(KafkaStarterUtils.java:46)

Chris Theodore Jayakumar

12/17/2021, 1:57 AM

we are not using any kafka config at the moment.. java version is Corretto-11.0.13.8.1… any help will be appreciated.

Xiang Fu

12/17/2021, 7:16 AM

Hmm, which command are you running ?

Chris Theodore Jayakumar

12/17/2021, 11:02 AM

Copy code

spark-submit --master "yarn"--class "org.apache.pinot.tools.admin.PinotAdministrator" s3://<bucket_name>/lib/pinot-all-0.9.0-jar-with-dependencies.jar LaunchDataIngestionJob -jobSpecFile /usr/local/pinot/jobs/<table_name>/job.yaml

abhinav wagle

01/14/2022, 6:49 PM

@Chris Theodore Jayakumar: Just following up here. I am facing a similar issue. any pointers on how you resolved this. Thanks!

Chris Theodore Jayakumar

01/17/2022, 1:20 AM

This was the error we faced when using EMR… we ended up with a dockerized solution in the end to get this working with a custom version of spark & hadoop

abhinav wagle

01/17/2022, 2:13 AM

Gotcha. Thanks. So you never got it working with EMR ?

Chris Theodore Jayakumar

01/17/2022, 2:13 AM

nope..

👍 1

Open in Slack

Previous Next