Dan Hill
10/07/2020, 1:44 AMCaused by: groovy.lang.MissingPropertyException: No such property: PINOT_DATE for class: SimpleTemplateScript1
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:53) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-8782e47b45c945cd02ccb8f06597b0ffa66a735a]
apiVersion: v1
kind: ConfigMap
metadata:
name: pinot-s3-data-config
data:
local_batch_job_spec.yaml: |-
executionFrameworkSpec:
...
inputDirURI: '<s3://promoted-event-logs/offlinepinot/dt=$JOB_DATE/>'
...
---
apiVersion: batch/v1
kind: Job
metadata:
name: pinot-populate-from-s3
spec:
template:
spec:
containers:
- name: pinot-populate-from-s3
image: apachepinot/pinot:0.6.0-SNAPSHOT-8782e47b4-20201006-jdk8
env:
- name: JOB_DATE
value: $(date -u +"%Y-%m-%d")
args: [
"LaunchDataIngestionJob",
"-jobSpecFile",
"/home/pinot/pinot-config/local_batch_job_spec.yaml"
]
volumeMounts:
- name: pinot-s3-data-config
mountPath: /home/pinot/pinot-config
restartPolicy: OnFailure
volumes:
- name: pinot-s3-data-config
configMap:
name: pinot-s3-data-config
backoffLimit: 100
Neha Pawar
Xiang Fu
LaunchDataIngestionJobCommand
provides an option of -values
to override the template contextargs: [
"LaunchDataIngestionJob",
"-jobSpecFile",
"/home/pinot/pinot-config/local_batch_job_spec.yaml"
"-values",
"yyyy=2020",
"mm=10"
]
-values jobDate=$JOB_DATE
in the command line and set inputDirURI: '<s3://promoted-event-logs/offlinepinot/dt=${jobDate}/>'
in your jobSpec fileDan Hill
10/07/2020, 2:40 AM