i am facing issue with the spark submit command for the cluster mode.. For the client mode i was able to run fine.. I am following the
https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark documentation… getting below error
Exception in thread “main” java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in scheme name at index 0: “s3:
command:
export PINOT_VERSION=0.11.0
export PINOT_DISTRIBUTION_DIR=
s3://roku-dea-dev/sand-box/suraj/apache-pinot-0.11.0-SNAPSHOT-bin
spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand --master yarn --deploy-mode cluster --conf “spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins” --conf “spark.driver.extraClassPath=/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-SNAPSHOT-shaded.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-SNAPSHOT-shaded.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-SNAPSHOT-shaded.jar” --conf “spark.executor.extraClassPath=/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-SNAPSHOT-shaded.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-SNAPSHOT-shaded.jar:/home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-SNAPSHOT-shaded.jar” --jars “${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar,${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-SNAPSHOT-shaded.jar,${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-s3/pinot-s3-${PINOT_VERSION}-SNAPSHOT-shaded.jar,${PINOT_DISTRIBUTION_DIR}/plugins/pinot-input-format/pinot-parquet/pinot-parquet-${PINOT_VERSION}-SNAPSHOT-shaded.jar” --files
s3://roku-dea-dev/sand-box/suraj/spark_job_spec.yaml local:///home/ssheshadri/suraj/clustercode/pinot/pinot-distribution/target/apache-pinot-0.11.0-SNAPSHOT-bin/apache-pinot-0.11.0-SNAPSHOT-bin/lib/pinot-all-${PINOT_VERSION}-SNAPSHOT-jar-with-dependencies.jar -jobSpecFile spark_job_spec.yaml