Hi Team, spark-submit ..... pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar -jobSpecFile spark_job_spec.yaml Can pinot-batch-ingestion jar not read jobSpecFile from S3/HDFS? currently it works if jobspecfile is present in local but errors out if I pass it in S3.

Yes, that is currently one of the limitations. Reason being we need configs from yaml file to access the S3 file system itself.

haha that’s interesting limitation :P but can’t we get to know the file system from file path itself ?

that is fine. What we need are the access/secret keys and other configurations from the

pinotFsSpec

in the yaml file

Can’t we make it part of other arguments ? Like having more than one argument to the spark submit command

i think there is value in keeping everything in one place. it makes it easier to document, and share, and compare etc.