Hi everyone , I am trying to run the spark ingestion job with apache pinot 0.8.0 in my own cluster setup. I am able to run the standalone job, but when am trying to run spark ingestion job it is giving me following error: java.lang.RuntimeException: Failed to create IngestionJobRunner instance for class - org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner I am using the below command to run the spark job: ${SPARK_HOME}/bin/spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand --master "local[2]" --deploy-mode client --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar" local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar -jobSpecFile ${PINOT_DISTRIBUTION_DIR}/examples/batch/transcript/transcript_local_jobspec.yaml I am also attaching screenshot of the error and my job spec file for better understanding. Could anyone please help with the same ?

What version of spark? If 3.x, may be try 2.x

If you’re running Pinot 0.8, then I think this is a known regression (from 0.7.1). See https://github.com/apache/pinot/issues/7393, which has some details of how @Kulbir Nijjer worked around this using Spark’s support for

dependencyJarDir

. There’s a PR (https://github.com/apache/pinot/pull/7400) to fix this, which works for Hadoop, haven’t tried with Spark.

It worked , I had to provide jars from plugin directory within apache-pinot-0.8.0-SNAPSHOT-bin folder