Apache Pinot

Hi. I am trying to submit a spark job for the offline batch ingestion process. I am running spark 2.2 and executors are being added and removed continuously without any work being done. Here is a gist of the log output <https://gist.github.com/drojas1/50556945f0491306dcf24c2ccd499446>
Can I get any advise about this issue?

The following is my spark submit command:
```/spark-home/spark/bin/spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand --conf spark.driver.userClassPathFirst=true --conf spark.executor.userClassPathFirst=true --conf 'spark.executor.extraJavaOptions==-Djava.io.tmpdir=/spark-home/spark/jtemp -Dlog4j2.configurationFile=/spark-home/pinot-resources/conf/pinot-ingestion-job-log4j2.xml' --conf 'spark.driver.extraJavaOptions==-Djava.io.tmpdir=/spark-home/spark/jtemp -Dlog4j2.configurationFile=/spark-home/pinot-resources/conf/pinot-ingestion-job-log4j2.xml' --properties-file /spark-home/resources/conf/spark.conf --files /spark-home/pinot-resources/pinot-job-spec.yaml --jars /spark-home/pinot-resources/plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/pinot-batch-ingestion-standalone-0.11.0-SNAPSHOT-shaded.jar,/spark-home/pinot-resources/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-0.11.0-SNAPSHOT-shaded.jar,/spark-home/pinot-resources/lib/pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar,/spark-home/pinot-resources/plugins/pinot-file-system/pinot-s3/pinot-s3-0.11.0-SNAPSHOT-shaded.jar,/spark-home/pinot-resources/plugins/pinot-input-format/pinot-parquet/pinot-parquet-0.11.0-SNAPSHOT-shaded.jar --driver-memory 114721M --driver-java-options '-Dspark.local.dir=/spark-home/spark -Dlog4j2.configurationFile=/spark-home/pinot-resources/conf/pinot-ingestion-job-log4j2.xml' --driver-class-path /spark-home/pinot-resources/plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/pinot-batch-ingestion-standalone-0.11.0-SNAPSHOT-shaded.jar,/spark-home/pinot-resources/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-0.11.0-SNAPSHOT-shaded.jar,/spark-home/pinot-resources/lib/pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar,/spark-home/pinot-resources/plugins/pinot-file-system/pinot-s3/pinot-s3-0.11.0-SNAPSHOT-shaded.jar,/spark-home/pinot-resources/plugins/pinot-input-format/pinot-parquet/pinot-parquet-0.11.0-SNAPSHOT-shaded.jar --executor-memory 114721M vig-spark-submit.XXXX.H9mrmRCl/pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar -jobSpecFile /spark-home/pinot-resources/pinot-job-spec.yaml -values 'inputDir="<s3://viglink-db-compressed-log/merchant_group/2022-07-20/>"' 'outputDir="<s3://ad-hoc-analysis/sukesh/pinot-offline-segments/2022-07-20/>"' 'tableName="merchant_group"' 'controllerURI="<https://pinot.internal.qa.sovrn.startree.cloud>"'```

```22/07/25 22:14:37 ERROR spark.SparkContext: Failed to add file:/mnt/persistent/spark/vig-spark-submit.XXXX.H9mrmRCl/pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar to Spark environment
java.lang.IllegalArgumentException: requirement failed: File pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar was already registered with a different path (old path = /spark-home/pinot-resources/lib/pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar, new path = /mnt/persistent/spark/vig-spark-submit.XXXX.H9mrmRCl/pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar```
Seems the same jar is registered twice