Oguzhan Mangir
04/02/2021, 1:53 PMjobSpecFile
config for reading job spec instead of local path?
${SPARK_HOME}/bin/spark-submit \\
--class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \\
--master "local[2]" \\
--deploy-mode client \\
--conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \\
--conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" \\
local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \\
-jobSpecFile ${PINOT_DISTRIBUTION_DIR}/examples/batch/airlineStats/sparkIngestionJobSpec.yaml
like;
-jobSpecFile <hdfs://bucket/pinot-specs/sparkIngestionJobSpec.yaml>
Mayank
LaunchDataIngestionJobCommand
seems to assume jobSpecFile is local.Mayank
Xiang Fu
Xiang Fu
Mayank
Oguzhan Mangir
04/03/2021, 2:26 PM