Hi, I am trying to create a Pinot Table that consumes the data from cross account Kinesis Streams. I have set environmental variables with credentials to get access to the cross account stream. However, I received an error in the corresponding pod that is trying to create the table: Error:

{
  "code": 500,
  "error": "org.apache.pinot.shaded.software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException: Stream cross-account-stream under account xxxxxxxx8241 not found. (Service: Kinesis, Status Code: 400, Request ID: c3ed6e9c-d25b-dccd-9873-aee2d3c8f651, Extended Request ID: 0/0WUQm7Y1TiHJj0Xr207jlzuDDZw8iv9NyFrUHvHz5WH8kRxc6xNO0lZDxOCgzy8HyBAOzHrxW8F0097tbWUqOp0fTnCJRw)"
}

YAML File:

apiVersion: v1
kind: ConfigMap
metadata:
  name: examples
  namespace: pinot-quickstart
data:
  CROSS_ACCESS_KEY_ID: "sample_key"
  CROSS_SECRET_ACCESS_KEY: "sample_secret_key"
  CROSS_SESSION_TOKEN: "sample_session_token"
  crossAccountEvents_realtime_table_config.json: |-
    {
      "tableName": "crossAccountEvents",
      "tableType": "REALTIME",
      "segmentsConfig": {
        "timeColumnName": "timeStampField",
        "retentionTimeUnit": "DAYS",
        "retentionTimeValue": "60",
        "schemaName": "crossAccountEvents",
        "replication": "1",
        "replicasPerPartition": "1"
      },
      "tenants": {},
      "tableIndexConfig": {
        "loadMode": "MMAP",
        "invertedIndexColumns": [
        ],
        "streamConfigs": {
          "streamType": "kinesis",
          "stream.kinesis.consumer.type": "lowlevel",
          "stream.kinesis.topic.name": "cross-account-stream",
          "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
          "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
          "realtime.segment.flush.threshold.time": "12h",
          "realtime.segment.flush.threshold.size": "100000",
          "stream.kinesis.consumer.prop.auto.offset.reset": "smallest",
          "region": "us-east-1",
          "shardIteratorType": "TRIM_HORIZON"
        }
      },
      "metadata": {
        "customConfigs": {}
      }
    }

  crossAccountEvents_schema.json: |-
    {
      "schemaName": "crossAccountEvents",
      "dimensionFieldSpecs": [
        {
          "name":"requestId",
          "dataType": "STRING",
          "defaultNullValue": ""
        },
        {
          "name": "accountId",
          "dataType": "STRING",
          "defaultNullValue": ""
        },
        {
          "name": "data",
          "dataType": "STRING",
          "defaultNullValue": ""
        }
      ],
      "dateTimeFieldSpecs": [
        {
          "name": "timeStampField",
          "dataType": "TIMESTAMP",
          "format": "1:MILLISECONDS:TIMESTAMP",
          "granularity": "1:MILLISECONDS"
        }
      ]
    }

---
apiVersion: batch/v1
kind: Job
metadata:
  name: cross-account-events-table-creation
  namespace: pinot-quickstart
spec:
  template:
    spec:
      containers:
        - name: cross-account-events-table-creation-json
          image: apachepinot/pinot:latest
          args: [ "AddTable", "-schemaFile", "/var/pinot/examples/crossAccountEvents_schema.json", "-tableConfigFile", "/var/pinot/examples/crossAccountEvents_realtime_table_config.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ]
          env:
            - name: JAVA_OPTS
              value: "-Xms4G -Xmx4G -Dpinot.admin.system.exit=true"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                configMapKeyRef:
                  key: CROSS_ACCESS_KEY_ID
                  name: examples
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                configMapKeyRef:
                  key: CROSS_SECRET_ACCESS_KEY
                  name: examples
            - name: AWS_SESSION_TOKEN
              valueFrom:
                configMapKeyRef:
                  key: CROSS_SESSION_TOKEN
                  name: examples
          volumeMounts:
            - name: examples
              mountPath: /var/pinot/examples
      restartPolicy: OnFailure
      volumes:
        - name: examples
          configMap:
            name: examples
  backoffLimit: 100

Can you please help me in resolving this error?

@Navina ^^

you can set the environment variables in your table config: https://docs.pinot.apache.org/configuration-reference/table#environment-variables-override

Oh, these environment variables are only available in Job pods and not server pods. Does that mean I need to export these environment variable to the server as well? Right now, I am just trying an hardcoded value in the config map