Pinot REALTIME Table Creation Fails with Strimzi Kafka TLS Authentication env: (Pinot helm chart: 0.3.4, Strimzi Kafka 3.7 both deployed on the same gke cluster) We are trying to create a REALTIME table in Pinot that connects to a Kafka cluster secured with TLS using certificates generated by Strimzi KafkaUser. The KafkaUser has full ACL rights (all operations on all resources). The issue appears to be during the table creation process when Pinot tries to fetch partition metadata from Kafka. I think this suggests that SSL connection is established but something fails during Kafka topic introspection. note: When I run the same configuration but with plain kafka connect (not tls) topic creation works fine. Error Details Controller Logs - Main Exception

2025/09/12 13:10:21.281 ERROR [PinotTableRestletResource] [grizzly-http-server-0] null
java.lang.RuntimeException
        at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:110)
        at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getNewPartitionGroupMetadataList(PinotLLCRealtimeSegmentManager.java:1174)
        at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getNewPartitionGroupMetadataList(PinotLLCRealtimeSegmentManager.java:1160)
        at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.setUpNewTable(PinotLLCRealtimeSegmentManager.java:381)
        at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.addTable(PinotHelixResourceManager.java:1822)
        at org.apache.pinot.controller.api.resources.PinotTableRestletResource.addTable(PinotTableRestletResource.java:255)
        [... stack trace continues ...]

2025/09/12 13:10:21.283 INFO [ControllerResponseFilter] [grizzly-http-server-0] Handled request from 127.0.0.1 POST <http://localhost:9000/tables>, content-type application/json status code 500 Internal Server Error

2025/09/12 13:10:21.283 INFO [PinotTaskManager] [ZkClient-EventThread-128-pinot-zookeeper:2181] Cleaning up task in scheduler for table testTable_REALTIME

I tried to misconfigure the SSL configuration, then I got this timeout error:

Caused by: org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
        at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchPartitionCount(KafkaStreamMetadataProvider.java:76)

Configuration Certificates are generated by Strimzi KafkaUser and converted to PKCS#12 format: -

user.p12

- Client keystore with user certificate and private key -

truststore.p12

- Truststore with Kafka cluster CA certificate - Password is read from

user.password

file Pinot Helm Configuration:

controller:
  extra:
    configs: |-
      pinot.set.instance.id.to.hostname=true
      controller.task.scheduler.enabled=true
      stream.kafka.broker.list=kafka-bootstrap:9093
      stream.kafka.consumer.factory.class.name=org.apache.pinot.plugin.stream.kafka.KafkaConsumerFactory
      security.protocol=SSL
      ssl.truststore.location=/opt/pinot/kafka-cert/truststore.p12
      ssl.truststore.password=changeit
      ssl.truststore.type=PKCS12
      ssl.keystore.location=/opt/pinot/kafka-cert/user.p12
      ssl.keystore.password=[read-from-file]
      ssl.keystore.type=PKCS12
      ssl.key.password=[read-from-file]
      ssl.endpoint.identification.algorithm=
...

Table Configuration

{
  "tableName": "testTable",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "DaysSinceEpoch",
    "timeType": "DAYS",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "7",
    "segmentPushType": "APPEND",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
    "replication": "1"
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "LowLevel",
      "stream.kafka.topic.name": "test-topic",
      "stream.kafka.broker.list": "kafka-bootstrap:9093",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaConsumerFactory",
      "security.protocol": "SSL",
      "ssl.truststore.location": "/opt/pinot/kafka-cert/truststore.p12",
      "ssl.truststore.password": "changeit",
      "ssl.truststore.type": "PKCS12",
      "ssl.keystore.location": "/opt/pinot/kafka-cert/user.p12",
      "ssl.keystore.password": "[read-from-file]",
      "ssl.keystore.type": "PKCS12",
      "ssl.key.password": "[read-from-file]",
      "ssl.endpoint.identification.algorithm": ""
    }
  }
}

Questions 1. Is Strimzi KafkaUser TLS authentication supported? I tried to follow the documentation examples at (https://docs.pinot.apache.org/manage-data/data-import/pinot-stream-ingestion/import-from-apache-kafka#use-kafka-partition-l[…]evel-consumer-with-ssl) 2. Configuration precedence: Should SSL configuration be in global Pinot config, streamConfigs, or both? I've tried both approaches. 3. Certificate format: Are PKCS#12 keystores the correct format? Strimzi generates PEM certificates that I convert using keytool. 4. Debugging: Are there specific logs or debug flags that could help identify if the SSL handshake is successful but authorization is failing? What I've Verified: - Kafka topic and KafkaUser exists and has correct permissions - Certificates are properly mounted in Pinot pods (

/opt/pinot/kafka-cert/

) - Schema creation works fine (

POST /schemas

succeeds) Any suggestions on proper Strimzi integration or debugging steps would be greatly appreciated!

Can you try

"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory",

1. pinot will pass all the corresponding kafka consumer configs, so it should be supported 2. should put those under

streamConfigs

in

tableconfig

Thanks @Mayank and @Xiang Fu that works..