Reposting for visibility: i am trying to enable iam auth for kafka using an msk cluster. I tried setting these properties:

SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL

SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION=/tmp/kafka.client.truststore.jks

SPRING_KAFKA_PROPERTIES_SASL_MECHANISM=AWS_MSK_IAM

SPRING_KAFKA_PROPERTIES_SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;

SPRING_KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

and also set the

KAFKA_BOOTSTRAP_SERVER

to the MSK public endpoint and the

KAFKA_ZOOKEEPER_CONNECT

in kafka_setup to the zookeper TLS endpoint in msk cluster. I did this within docker.env inside gms, mce-consumer, mae-consumer and kafka-setup, but its not connecting to the msk cluster. I am getting this error:

Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: No LoginModule found for software.amazon.msk.auth.iam.IAMLoginModule

kafka-setup               |     at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:172)

kafka-setup               |     at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)

kafka-setup               |     at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)

kafka-setup               |     at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)

kafka-setup               |     at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)

Hi Sarat, have you walked through our Kafka ingestion guide here? https://datahubproject.io/docs/how/kafka-config

Hi Paul, I think Sarat is not trying to ingest from Kafka, but more trying to connect his DataHub backend to Kafka to run DataHub! I believe this is because we do not have the AWS classes on the classpath

@astonishing-answer-96712 @big-carpet-38439 is correct. I am trying to connect Datahub backend to an AWS MSK cluster and enable IAM auth for this MSK cluster