Hi All, i am working on spark ingestion job to push previous date data everyday and it is working fine locally using this command - bin/pinot-ingestion-job.sh -jobSpecFile ${PINOT_DIR}/ingestionJobSpec.yaml -values date=

date -v-1d +%F

where ' date is set under includeFilePattern parameter -includeFileNamePattern: 'glob:**/{date}/*.avro'. While executing spark submit job with this command - $SPARK_HOME/bin/spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand --master "local[2]" \ --deploy-mode client --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins \ -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \ --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar/Users/deemish2/apache pinot 0.8.0 bin/plugins/pinot batch ingestion/pinot batch ingestion spark/pinot batch ingestion spark 0.8.0 shaded.jar/Users/deemish2/apache-pinot-0.8.0-bin/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-0.8.0-shaded.jar" \ local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \ -jobSpecFile ${PINOT_DIR}/SparkingestionJobSpec.yaml -values date=

date -v-1d +%F

It gives error like - Caused by: java.lang.IllegalArgumentException: Positive number of partitions required. It looks like this argument - -values date=

date -v-1d +%F

. works only with bin/pinot-ingestion.sh. Please help to execute this spark ingestion job to push previous date data in pinot

@Deepak Mishra Were you able to solve this issue and pass argument to yaml file?