hello everyone, does any have any idea on this ? ....
# general
hello everyone, does any have any idea on this ? . i' m using latest release version of pinot (0.7.1). while doing the spark batchIngestion, its is throwing this. thanks
Copy code
Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/VoidFunction
	at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
	at java.base/java.lang.Class.getConstructor0(Class.java:3342)
	at java.base/java.lang.Class.getConstructor(Class.java:2151)
	at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:295)
	at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:264)
	at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:245)
	at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:135)
I think pinot-spark requires spark version plugins
you need to mount your spark libs to the container and set it as part of the classpath
typically people use the library directly in their spark cluster
true. i ran this command
bin/pinot-admin.sh LaunchDataIngestionJob  -jobSpecFile /data_ssd/spark-ingestion/spark-ingestion.yml
installed spark. added the SPARK_HOME and spark sbin to path
will that be suffice ?
should i try it on emr ? any recomendations on spark version and scala version @User
It should work with Standalone spark installed locally too, I assume Spark was installed after you saw the initial error ? If yes please try it again, basically required jar
is included in <SPARK_HOME>/jars and running spark-submit should set required classpath variables. EMR can also be used if you have a running Spark cluster and subsequently Spark-Submit command you run in Pinot will require changes to
settings accordingly.
👍 1
im using this image
. i tried it again and trying to run the examples one.
message has been deleted
You can download the jar and use it
The image doesn’t contain spark core libs
Image it self is a separated container which has no access to your environment
So you need to mount all your spark classpath to it and update the classpath
i installed spark in the container