I tried to build the source code with mvn option -

Djdk.version=8.

(https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally) i am facing errror Caused by: java.lang.ClassNotFoundException: org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner.. the jar built doesnt have this class. Any suggestions here?

What version of spark?

2.4.6

Are you following this: https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark

why java 8 😞

our EC2 cluster in dev has java 8 installed.. so was trying this option out… i also tried to install java 11 on the cluster and do a spark submit but faced another issue of : Exception in thread “main” java.lang.VerifyError: Bad return type Exception Details: Location: org/apache/hadoop/hdfs/DFSClient.getQuotaUsage(Ljava/lang/String;)Lorg/apache/hadoop/fs/QuotaUsage; @157: areturn

Hi The docs are not updated for master build. In the latest master, we have moved spark plugins to another directory. Replace the following in your command

${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark/pinot-batch-ingestion-spark-${PINOT_VERSION}-shaded.jar

with

${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar

@Kartik Khare I faced below 3 issues after the update on the command.. can you please suggest. 1. Error: Unrecognized option: -Dplugins.include=pinot-s3,pinot-parquet 2)Error: Unrecognized option: -Dlog4j2.configurationFile=/home/ssheshadri/suraj/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/conf/pinot-ingestion-job-log4j2.xml” 3) Can’t construct a java object for tag:yaml.org,2002:org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec; exception=Class not found: org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec in ‘string’, line 1, column 1: executionFrameworkSpec:

Hi. you can remove the first 2 as they are completely optional. For the 3rd point, I would need to understand your cluster setup. For some reason

pinot-all

jar is not available on the classpath

The issue was resolved.. thank you @Kartik Khare.. Below fixes were done: path was inaccurate in original command for ${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar path was wrong in config file updated to :

segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentMetadataPushJobRunner'

Final working command: export PINOT_VERSION=0.11.0 export PINOT_DISTRIBUTION_DIR=/home/ssheshadri/suraj/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand --master local --deploy-mode client --conf “spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dplugins.include=pinot-s3,pinot-parquet -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml” --conf “spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-SNAPSHOT-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-SNAPSHOT-shaded.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-SNAPSHOT-shaded.jar” --conf “spark.executor.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-SNAPSHOT-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-SNAPSHOT-shaded.jar:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-SNAPSHOT-shaded.jar” local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar -jobSpecFile /home/ssheshadri/suraj/spark_job_spec.yaml