@Kishore G is kubernetes configmap and job a good way to create tables or there is another way best recommended?

You can import secrets to env variables and mount table request template using configmap

I was trying to create a

jaas.conf

as secret mount and set the

-Djava.security.auth.login.config=/etc/kafka/secrets/jaas.conf

environment variable so Pinot could read the file... was something like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cfmp
  namespace: pinot
data:
  realtime.json: |-
    {   
        ...
        # REALTIME table
    }
        

  schema.json: |-
    {
        # schema
    }
---
apiVersion: batch/v1
kind: Job
metadata:
  name: job
  namespace: pinot
spec:
  template:
    spec:
      containers:
        - name: pinot-job
          image: apachepinot/pinot:latest
          args: [ "AddTable", "-schemaFile", "/var/pinot/cfmp/schema.json", "-tableConfigFile", "/var/pinot/cfmp/realtime.json", "-controllerHost", "pinot-controller", "-controllerPort", "9000", "-exec" ]
          env:
            - name: JAVA_OPTS
              value: "-Xms4G -Xmx4G -Dpinot.admin.system.exit=true -Djava.security.auth.login.config=/etc/kafka/secrets/jaas.conf"  
          volumeMounts:
            - name: cfmp
              mountPath: /var/pinot/examples              
            - name: kafka-secret
              mountPath: /etc/kafka/secrets
              subPath: rest_jaas.conf
      restartPolicy: Never
      volumes:
        - name: cfmp
          configMap:
            name: cfmp
        - name: kafka-secret
          secret:
            secretName: kafka-secret          
  backoffLimit: 0

Note that

-Djava.security.auth.login.config=/etc/kafka/secrets/jaas.conf"

this kind of JVM opts has to be set at startup time

which is not the right way, cause you may want to connect to multiple kafka

@Xiang Fu Hi there, even with it's drawback I wanna try to auth hiddin the username and password. I've tried the following: • created a jaas.conf:

KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="foo"
    password="bar";
};

• on the controller, passed the path of the file:

"-Djava.security.auth.login.config=/etc/kafka/secrets/jaas.conf"

• set table conf as following:

[...]
    "tableIndexConfig": {
        "loadMode": "MMAP",
        "streamConfigs": {
            "streamType": "kafka",
            "stream.kafka.topic.name": "test",
            "stream.kafka.consumer.type": "lowlevel",
            "stream.kafka.consumer.prop.auto.offset.reset": "7d",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
            "stream.kafka.broker.list": "my-confluent-address:9092",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
            "stream.kafka.decoder.prop.basic.auth.credentials.source": "USER_INFO",
            "sasl.mechanism": "PLAIN",
            "security.protocol": "SASL_PLAINTEXT",
            "sasl.enabled.mechanisms": "PLAIN",
            "realtime.segment.flush.threshold.rows": "50000",
            "realtime.segment.flush.threshold.time": "3h",
            "realtime.segment.flush.threshold.segment.size": "150M"                   
        }
    }
[...]

Now when when I add the table I receive the timeout error:

{
  "code": 500,
  "error": "org.apache.pinot.spi.stream.TransientConsumerException: org.apache.pinot.shaded.org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata"
}

Do you have any clue what am I messing?

Another try is to load this jaas content as an environment variable, then use table config override: https://docs.pinot.apache.org/configuration-reference/table#environment-variables-override

note, you need this for all pinot controller/servers to ensure the connectivity

@Xiang Fu do you know other way I can achieve this? Not showing creds on ui after the table is created?

Actually I used the config "*queryConsoleOnlyView*" to achieve what I was looking for

thanks