Hello, I work on connecting to an AWS MSK cluster with IAM authentication method and I failed with the following error - the full error message can be found in error.txt.

...

Caused by: org.apache.flink.kafka.shaded.org.apache.kafka.common.KafkaException: java.lang.ClassCastException: class software.amazon.msk.auth.iam.IAMClientCallbackHandler cannot be cast to class org.apache.flink.kafka.shaded.org.apache.kafka.common.security.auth.AuthenticateCallbackHandler (software.amazon.msk.auth.iam.IAMClientCallbackHandler and org.apache.flink.kafka.shaded.org.apache.kafka.common.security.auth.AuthenticateCallbackHandler are in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @7ed85b97)
        at org.apache.flink.kafka.shaded.org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
        at org.apache.flink.kafka.shaded.org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
        at org.apache.flink.kafka.shaded.org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
        at org.apache.flink.kafka.shaded.org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at org.apache.flink.kafka.shaded.org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:513)
        ... 13 more
Caused by: java.lang.ClassCastException: class software.amazon.msk.auth.iam.IAMClientCallbackHandler cannot be cast to class org.apache.flink.kafka.shaded.org.apache.kafka.common.security.auth.AuthenticateCallbackHandler (software.amazon.msk.auth.iam.IAMClientCallbackHandler and org.apache.flink.kafka.shaded.org.apache.kafka.common.security.auth.AuthenticateCallbackHandler are in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @7ed85b97)
        at org.apache.flink.kafka.shaded.org.apache.kafka.common.network.SaslChannelBuilder.createClientCallbackHandler(SaslChannelBuilder.java:305)
        at org.apache.flink.kafka.shaded.org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:148)
        ... 17 more

I tried both adding multiple JAR files and a single fat jar file as I plan to deploy it to Kinesis Data Analytics later. Both raise the same error and the POM file for the fat jar can be found in pom.xml. Also The relevant app configuration can be found in src.py. Can you please inform me how to fix this error?

The MSK IAM Auth library is not compatible with the SQL Kafka connector (Uber jar) due to shade relocation. Change

flink-sql-connector-kafka

to

flink-connector-kafka

Thank you @Danny Cranmer! Let me check it out.

@Jaehyeon Kim did you ever get this working? I'm using the non-sql jars and still getting this error

Hi Brian Yes, I was able to manage it. I wrote a post about it. Hope it helps. https://jaehyeon.me/blog/2023-08-28-getting-started-with-pyflink-on-aws-part-2/

Thanks so much! I've been dealing with a lot of msk iam issues. I feel your blog will help a lot