Hello , we are using spark batch ingestion job to push data into pinot offline table using pinot-0.8.0 , we are getting this kind of exception -
Caused by: groovy.lang.MissingPropertyException: No such property: date for class: SimpleTemplateScript1\n\tat org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:66)
x
Xiang Fu
10/11/2021, 6:02 PM
i think your ingestion job spec has groovy templates but propeties are not set
d
Deepak Mishra
10/12/2021, 6:11 AM
we are not using groovy template in ingestion job spec file
x
Xiang Fu
10/12/2021, 2:58 PM
Hmm, how do you run the ingestion job
d
Deepak Mishra
10/12/2021, 3:26 PM
I am running ingestion job using spark-submit
$SPARK_HOME/bin/spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \
--master "local" --deploy-mode client --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins \
-Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \
--conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar
:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/pinot-batch-ingestion-spark-${PINOT_VERSION}-shaded.jar
:${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-${PINOT_VERSION}-shaded.jar" \
${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \
-jobSpecFile ${PINOT_DIR}/SparkingestionJobSpec.yaml