Hi community, I'm trying to integrate Apache Pinot...
# troubleshooting
s
Hi community, I'm trying to integrate Apache Pinot (running on Amazon EKS) with Amazon MSK (IAM-auth enabled). My goal is to securely consume Kafka topics using IAM-based authentication. Here’s what I’ve tried so far: • Stored my MSK authentication credentials (secret,) as Kubernetes Secrets in EKS. • Mounted these secrets into my Pinot pods as environment variables. • Updated my table config to use port 9096 (SCRAM) and set the following Kafka consumer properties Despite this, Pinot is unable to connect to the MSK broker and fails during table creation when I use broker port as 9096 , but table creation is achieved when I use port 9092. I would really appreciate any guidance: Has anyone successfully set up MSK IAM authentication with Pinot? Thanks in advance for your help!
m
That implies auth is not working right? I'd start with checking if you are missing table config setup, and then the IAM role setting.
s
Yes , auth is not working . Let me explain what steps I followed for that. I have stored secrets in secrets manager for MSK Cluster ,and created secrets in EKS using yaml.I than fetched those secrets in env variable in server pod and controller pod ,and updated to able config file to use SCRAM .Here are the lines that I added in table config file- "stream.kafka.consumer.prop.security.protocol": "SASL_SSL", "stream.kafka.consumer.prop.sasl.mechanism": "SCRAM-SHA-512", "stream.kafka.consumer.prop.sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"${KAFKA_USERNAME}\" password=\"${KAFKA_PASSWORD}\";" Still it's not working .