Hello, I’m having some trouble creating a table in an EKS cluster when I use environment variable overrides. Here’s an abridged version of the YAML I’m attempting to apply:

apiVersion: v1
kind: ConfigMap
metadata:
  name: test-dmp-with-json-config-map
  namespace: pinot
data:
  test_dmp_realtime_table_config.json: |-
    {
      ...
        "streamConfigs": {
          ...
          "stream.kafka.broker.list": "${CONFLUENT_BOOTSTRAP_SERVER}",
          "sasl.mechanism": "PLAIN",
          "sasl.jaas.config": "${CONFLUENT_JAAS_CONFIG}",
          "security.protocol": "SASL_SSL",
          ...
        }
      },
      ...
    }

  test_dmp_realtime_schema.json: |-
    {
      ...
    }
---
apiVersion: batch/v1
kind: Job
metadata:
  name: test-dmp-with-json-job
  namespace: pinot
spec:
  template:
    spec:
      containers:
        - name: pinot-add-test-dmp-json
          image: apachepinot/pinot:latest
          args: [ "AddTable", "-schemaFile", "/var/pinot/test-dmp-with-json/test_dmp_realtime_schema.json", "-tableConfigFile", "/var/pinot/test-dmp-with-json/test_dmp_realtime_table_config.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ]
          env:
            - name: JAVA_OPTS
              value: "-Xms4G -Xmx4G -Dpinot.admin.system.exit=true"
            - name: CONFLUENT_BOOTSTRAP_SERVER
              valueFrom:
                secretKeyRef:
                  name: confluent-credentials
                  key: confluent-bootstrap-server-sandbox
                  optional: false
            - name: CONFLUENT_JAAS_CONFIG
              valueFrom:
                secretKeyRef:
                  name: confluent-credentials
                  key: confluent-jaas-config-sandbox
                  optional: false
          volumeMounts:
            - name: test-dmp-with-json
              mountPath: /var/pinot/test-dmp-with-json
      restartPolicy: OnFailure
      volumes:
        - name: test-dmp-with-json
          configMap:
            name: test-dmp-with-json-config-map
  backoffLimit: 100

The error I’m getting in the created Pod is

{"code":500,"error":"Unable to apply environment variables on json config class [org.apache.pinot.spi.config.table.TableConfig]."}

I’m confident it has nothing to do with pulling the environment variables from a Secret, since I also tried hardcoding the credentials. Does anyone know what’s going wrong? Am I not putting the environment variables in the correct space?

Ah cool. Glad you figured it out.

@James Kelleher I am also facing the same issue, can you please post what changes you have made to the code? How can we share environment variables across all the pods? I am getting the same error, even when I hardcode the value