https://pinot.apache.org/ logo
d

Dan Hill

10/07/2020, 1:44 AM
I have a simple k8 job that runs LaunchDataIngestionJob. Is it possible to use an environment variable in the batch job spec? I'm trying to pass a date parameter into the input path. Here's the error and example spec.
Copy code
Caused by: groovy.lang.MissingPropertyException: No such property: PINOT_DATE for class: SimpleTemplateScript1
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:53) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-8782e47b45c945cd02ccb8f06597b0ffa66a735a]
Copy code
apiVersion: v1
kind: ConfigMap
metadata:
  name: pinot-s3-data-config
data:
  local_batch_job_spec.yaml: |-
    executionFrameworkSpec:
    ...
    inputDirURI: '<s3://promoted-event-logs/offlinepinot/dt=$JOB_DATE/>'
    ...

---
apiVersion: batch/v1
kind: Job
metadata:
  name: pinot-populate-from-s3
spec:
  template:
    spec:
      containers:
        - name: pinot-populate-from-s3
          image: apachepinot/pinot:0.6.0-SNAPSHOT-8782e47b4-20201006-jdk8
          env:
            - name: JOB_DATE
              value: $(date -u +"%Y-%m-%d")
          args: [
            "LaunchDataIngestionJob",
            "-jobSpecFile",
            "/home/pinot/pinot-config/local_batch_job_spec.yaml"
          ]
          volumeMounts:
            - name: pinot-s3-data-config
              mountPath: /home/pinot/pinot-config
      restartPolicy: OnFailure
      volumes:
        - name: pinot-s3-data-config
          configMap:
            name: pinot-s3-data-config
  backoffLimit: 100
n

Neha Pawar

10/07/2020, 1:55 AM
@Xiang Fu I remember you had introduced templates to pass variables. Can that help here?
x

Xiang Fu

10/07/2020, 2:01 AM
the
LaunchDataIngestionJobCommand
provides an option of
-values
to override the template context
k8s scripts and add that
Copy code
args: [
            "LaunchDataIngestionJob",
            "-jobSpecFile",
            "/home/pinot/pinot-config/local_batch_job_spec.yaml"
            "-values",
            "yyyy=2020",
            "mm=10"
          ]
I think you need to append
-values jobDate=$JOB_DATE
in the command line and set
inputDirURI: '<s3://promoted-event-logs/offlinepinot/dt=${jobDate}/>'
in your jobSpec file
d

Dan Hill

10/07/2020, 2:40 AM
Thanks! I'll try that