I have a simple k8 job that runs LaunchDataIngestionJob. Is it possible to use an environment variable in the batch job spec? I'm trying to pass a date parameter into the input path. Here's the error and example spec.

Caused by: groovy.lang.MissingPropertyException: No such property: PINOT_DATE for class: SimpleTemplateScript1
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:53) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-8782e47b45c945cd02ccb8f06597b0ffa66a735a]

apiVersion: v1
kind: ConfigMap
metadata:
  name: pinot-s3-data-config
data:
  local_batch_job_spec.yaml: |-
    executionFrameworkSpec:
    ...
    inputDirURI: '<s3://promoted-event-logs/offlinepinot/dt=$JOB_DATE/>'
    ...

---
apiVersion: batch/v1
kind: Job
metadata:
  name: pinot-populate-from-s3
spec:
  template:
    spec:
      containers:
        - name: pinot-populate-from-s3
          image: apachepinot/pinot:0.6.0-SNAPSHOT-8782e47b4-20201006-jdk8
          env:
            - name: JOB_DATE
              value: $(date -u +"%Y-%m-%d")
          args: [
            "LaunchDataIngestionJob",
            "-jobSpecFile",
            "/home/pinot/pinot-config/local_batch_job_spec.yaml"
          ]
          volumeMounts:
            - name: pinot-s3-data-config
              mountPath: /home/pinot/pinot-config
      restartPolicy: OnFailure
      volumes:
        - name: pinot-s3-data-config
          configMap:
            name: pinot-s3-data-config
  backoffLimit: 100

@Xiang Fu I remember you had introduced templates to pass variables. Can that help here?

the

LaunchDataIngestionJobCommand

provides an option of

-values

to override the template context

Thanks! I'll try that