2022/02/08 11:48:43.990 INFO [SecurityManager] [ma...
# troubleshootingr
Ravishankar Nair
02/08/2022, 4:54 PM2022/02/08 114843.990 INFO [SecurityManager] [main] SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ravishankar); groups with view permissions: Set(); users with modify permissions: Set(ravishankar); groups with modify permissions: Set()
Exception in thread "main" java.lang.NoSuchMethodException: org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main([Ljava.lang.String;)
at java.base/java.lang.Class.getMethod(Class.java:2108)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2022/02/08 114844.084 INFO [ShutdownHookManager] [Thread-0] Shutdown hook called
a
Aditya
02/08/2022, 6:18 PMThere is some change in class name
Check out this thread
https://apache-pinot.slack.com/archives/CDRCA57FC/p1641806567053000?thread_ts=1641806567.053000&cid=CDRCA57FC
Aditya
02/08/2022, 6:19 PMCorrect class name is LaunchDataIngestionJob
r
Ravishankar Nair
02/08/2022, 6:51 PMTHanks a lor @Aditya. It works. However one small issue- can you guide why am I getting java.lang.RuntimeException java.io.IOException: Failed to create directory: pinot-plugins-dir-0/plugins. THanks a lot...
Ravishankar Nair
02/08/2022, 6:52 PMDO I need to disable ACLs in S3 ?
a
Aditya
02/08/2022, 7:17 PMNot sure about this error. I only tried running spark job in local mode
The job extracted all plugins in the working directory. It shouldn't be related to s3.
Does the user id which is submitting this job have write access to the working directory?
r
Ravishankar Nair
02/08/2022, 7:24 PM@Aditya yes definitely. These segments are created in S3 right?
x
Xiang Fu
02/08/2022, 7:30 PM
do you have more stacktrace
Xiang Fu
02/08/2022, 7:30 PM
this seems to be the issue of plugin directory copy
r
Ravishankar Nair
02/08/2022, 7:32 PM@Xiang Fu Here you go:
Ravishankar Nair
02/08/2022, 7:32 PMat org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.foreach(RDD.scala:970)
at org.apache.spark.api.java.JavaRDDLike$class.foreach(JavaRDDLike.scala:351)
at org.apache.spark.api.java.AbstractJavaRDDLike.foreach(JavaRDDLike.scala:45)
at org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner.run(SparkSegmentGenerationJobRunner.java:246)
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:146)
... 25 more
Caused by: java.lang.RuntimeException: java.io.IOException: Failed to create directory: pinot-plugins-dir-0/plugins
at org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner$1.call(SparkSegmentGenerationJobRunner.java:267)
at org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner$1.call(SparkSegmentGenerationJobRunner.java:246)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
Ravishankar Nair
02/08/2022, 7:33 PMSome more:
Ravishankar Nair
02/08/2022, 7:33 PMat org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0 (TID 2, localhost, executor driver): java.lang.RuntimeException: java.io.IOException: Failed to create directory: pinot-plugins-dir-0/plugins
at org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner$1.call(SparkSegmentGenerationJobRunner.java:267)
at org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner$1.call(SparkSegmentGenerationJobRunner.java:246)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$foreach$1.apply(JavaRDDLike.scala:351)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
x
Xiang Fu
02/08/2022, 7:36 PM
I see. Pinot spark job internally requires to create a directory to load pinot jars, do you know if you spark worker has permission to operate on local disk?
Copy code
java.io.IOException: Failed to create directory: pinot-plugins-dir-0/pluginsr
Ravishankar Nair
02/08/2022, 7:39 PMNever needed so, as I have many spark code running on my local. Is there any staging directory I should configure? I saw a parameter in jobspec sample to configure staging
x
Xiang Fu
02/08/2022, 8:00 PM
stagingDir is for temp data storage
Xiang Fu
02/08/2022, 8:00 PM
can you try to put the plugin libs into your classpath and remove the pluginDir config when you launch the job
r
Ravishankar Nair
02/08/2022, 8:04 PMNo luck, then the first error is coming StartKafkaCOmmand....
Ravishankar Nair
02/08/2022, 8:05 PMI can see that when I run the job, Spark is creating a directory, even if I run as sudo, no permission...
Ravishankar Nair
02/08/2022, 8:06 PMcan I specify a user when running spark-submit?
Ravishankar Nair
02/08/2022, 8:08 PMat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Failed to create directory: pinot-plugins-dir-0/plugins/pinot-minion-tasks
at org.apache.pinot.common.utils.TarGzCompressionUtils.untar(TarGzCompressionUtils.java:133)
Ravishankar Nair
02/08/2022, 8:08 PMIt must be from pinot.
Ravishankar Nair
02/08/2022, 8:13 PMif (entry.isDirectory()) {
if (!outputFile.getCanonicalPath().startsWith(outputDirCanonicalPath)) {
throw new IOException(String
.format("Trying to create directory: %s outside of the output directory: %s", outputFile, outputDir));
}
if (!outputFile.isDirectory() && !outputFile.mkdirs()) {
throw new IOException(String.format("Failed to create directory: %s", outputFile));
}
Ravishankar Nair
02/08/2022, 8:13 PMTarGzCompressionUtils.java from Pinot
Ravishankar Nair
02/08/2022, 8:22 PM@Xiang Fu I see that you committed the code SegmentGenerationUtils which uses this directory.
x
Xiang Fu
02/08/2022, 8:24 PM
yes, it’s using java file api
Xiang Fu
02/08/2022, 8:24 PM
so pretty weird that this can fail
r
Ravishankar Nair
02/08/2022, 8:25 PMEVen I changed sudo chmod recursively and then re execute, still its failing
x
Xiang Fu
02/08/2022, 8:25 PM
hmm
r
Ravishankar Nair
02/08/2022, 8:25 PMI am using Mac. DO I need to disable anything
x
Xiang Fu
02/08/2022, 8:25 PM
oh? so you start a local spark cluster?
r
Ravishankar Nair
02/08/2022, 8:26 PMyes
Ravishankar Nair
02/08/2022, 8:26 PMThats the issue?
x
Xiang Fu
02/08/2022, 8:36 PM
not really
Xiang Fu
02/08/2022, 8:36 PM
which spark version are you using?
r
Ravishankar Nair
02/08/2022, 8:36 PM2.4.8
x
Xiang Fu
02/08/2022, 8:37 PM
have you read this: https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark
r
Ravishankar Nair
02/08/2022, 8:40 PMLet me go through, thanks
Ravishankar Nair
02/08/2022, 8:40 PMI have seen similar one, another URL in Pinot site
x
Xiang Fu
02/08/2022, 9:21 PM
This is the last time I ran the spark-submit:
Copy code
export PINOT_VERSION=0.7.0-SNAPSHOT
export PINOT_DISTRIBUTION_DIR=${PINOT_ROOT_DIR}/pinot-distribution/target/apache-pinot-incubating-${PINOT_VERSION}-bin/apache-pinot-incubating-${PINOT_VERSION}-bin
cd ${PINOT_DISTRIBUTION_DIR}
${SPARK_HOME}/bin/spark-submit
--class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand
--deploy-mode client
--conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml"
--conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar"
local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar
-jobSpecFile /Users/xiangfu/temp/pinot/pinot-s3-test/segmentMetadataSparkIngestionJobSpec.yamlXiang Fu
02/08/2022, 9:23 PM
I think the spark submit is now:
Copy code
${SPARK_HOME}/bin/spark-submit
--class org.apache.pinot.tools.admin.PinotAdministrator \\
--deploy-mode client \\
--conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \\
--conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" \\
local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \\
LaunchDataIngestionJobCommand \\
-jobSpecFile /Users/xiangfu/temp/pinot/pinot-s3-test/segmentMetadataSparkIngestionJobSpec.yamlXiang Fu
02/09/2022, 12:57 AM
I do see some issue with spark combining jdk11, scala, jackson, will take another look
r
Ravishankar Nair
02/09/2022, 1:07 AMOhh, it will not work now?
Ravishankar Nair
02/09/2022, 2:31 AMReally, please help. SOmeone needs to look into this. It does not work with Spark latest versions. With Spark 2.4 also, version 0.9.3 is giving weird issues
Ravishankar Nair
02/09/2022, 3:15 AMOK, One clue: I manually untarred the plugin.tar.gz to pinot-plugins-dir-0, now it works. So you may want to check why the code itself is unable to untar.
Ravishankar Nair
02/09/2022, 3:49 AM@Xiang Fu After successful ingestion, when I query airlineStats table, no rows returned. Where did I make a mistake? Result: {"exceptions":[],"numServersQueried":0,"numServersResponded":0,"numSegmentsQueried":0,"numSegmentsProcessed":0,"numSegmentsMatched":0,"numConsumingSegmentsQueried":0,"numDocsScanned":0,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numGroupsLimitReached":false,"totalDocs":0,"timeUsedMs":0,"offlineThreadCpuTimeNs":0,"realtimeThreadCpuTimeNs":0,"segmentStatistics":[],"traceInfo":{},"minConsumingFreshnessTimeMs":0,"numRowsResultSet":0}
x
Xiang Fu
02/09/2022, 3:59 AM
Hmm, then I feel no data are really ingested
Xiang Fu
02/09/2022, 4:00 AM
Since you are on local Mac. Can you just try with the standalone ingestion job ?
r
Ravishankar Nair
02/10/2022, 8:53 PM@Xiang Fu All perfect. DO you mind me writing an article in medium on this? Can you please review and give your comments? I would like @Aditya also to be part of it. Let the community be benefitted
x
Xiang Fu
02/10/2022, 9:53 PM
Sounds good
r
Ravishankar Nair
02/10/2022, 10:46 PMCan I have your email pls
Ravishankar Nair
02/10/2022, 10:46 PMOr pls send a mail to ravishankar.nair@gmail.com
12 Views