Apache Pinot

Hi team, I’m trying to connect AWS MSK - IAM authentication in Pinot console for realtime streaming purpose and facing connectivity issues. Most probably due to configuration issue. Do we have any example config for MSK IAM authentication? Thanks

Hi gopi, for MSK you don’t need to do anything special, just mention the ssl configs that you would use with normal kafka console consumer in the pinot stream configs as well



Thanks for the clarification <@U011R2VMS4S>.  Will try it out

Hi <@U011R2VMS4S>, I got the required configs to be added from this link <https://github.com/aws/aws-msk-iam-auth?tab=readme-ov-file#configuring-a-kafka-client-to-use-aws-iam-with-sasl-oauthbearer-mechanism>

Getting error and It looks like we need to add aws msk jar to kafka lib directory. Can you advise where to get kafka lib path  inside Pinot?

`"<http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.kafka.common.config.ConfigException: Invalid value software.amazon.msk.auth.iam.IAMOAuthBearerLoginCallbackHandler for configuration sasl.login.callback.handler.class: Class software.amazon.msk.auth.iam.IAMOAuthBearerLoginCallbackHandler could not be found."`

interesting, yeah for OUTHBEARER mechanism you will need a jar, you can copy it in pinot lib directory and restart controller and server


OR

you can switch MSK to use SASL plaintxt

Hi, getting this error after adding the jar in lib directory.  Looks like pinot is using shaded kafka lib and not compatible. Can you check please advise on this?

```Caused by: org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: java.lang.ClassCastException: class software.amazon.msk.auth.iam.IAMClientCallbackHandler cannot be cast to class org.apache.pinot.shaded.org.apache.kafka.common.security.auth.AuthenticateCallbackHandler (software.amazon.msk.auth.iam.IAMClientCallbackHandler and org.apache.pinot.shaded.org.apache.kafka.common.security.auth.AuthenticateCallbackHandler are in unnamed module of loader 'app')```

Also, trying to use SASL- SCRAM authentication and getting the error. Looks like SCRAM-SHA-512 mechanism is not enabled in server. Can you advise here?

```Caused by: org.apache.pinot.shaded.org.apache.kafka.common.errors.UnsupportedSaslMechanismException: Client SASL mechanism 'SCRAM-SHA-512' not enabled in the server, enabled mechanisms are [OAUTHBEARER, AWS_MSK_IAM]```