Hello , we are using spark batch ingestion job to ...
# troubleshooting
d
Hello , we are using spark batch ingestion job to push data into pinot offline table using pinot-0.8.0 , we are getting this kind of exception - Caused by: groovy.lang.MissingPropertyException: No such property: date for class: SimpleTemplateScript1\n\tat org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:66)
x
i think your ingestion job spec has groovy templates but propeties are not set
d
we are not using groovy template in ingestion job spec file
x
Hmm, how do you run the ingestion job
d
I am running ingestion job using spark-submit $SPARK_HOME/bin/spark-submit --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \ --master "local" --deploy-mode client --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins \ -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \ --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar :${PINOT_DISTRIBUTION_DIR}/plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/pinot-batch-ingestion-spark-${PINOT_VERSION}-shaded.jar :${PINOT_DISTRIBUTION_DIR}/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-${PINOT_VERSION}-shaded.jar" \ ${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \ -jobSpecFile ${PINOT_DIR}/SparkingestionJobSpec.yaml
SparkingestionJobSpec.yaml
x
you have this
Copy code
includeFileNamePattern: 'glob:**/{date}/*.json'
yaml will try to replace it with your date
but you are not giving the param