Hi there! I am trying to setup Kafka using MSK and IAM authentication. It's working fine for most services: • actions is using SCRAM-SHA-512, but that's okay for now. Any plans to support MSK IAM here? • gms, mae, mce and the schema registry are working fine with MSK IAM • What I cannot solve is telling the datahub-frontend to use MSK IAM. I have configured it like this:

- name: KAFKA_BOOTSTRAP_SERVER
  value: "XXXXXXXXXXXXXXXX:9098,YYYYYYYYYYYYYYYYY:9098"
- name: KAFKA_PROPERTIES_SECURITY_PROTOCOL
  value: "SASL_SSL"
- name: KAFKA_PROPERTIES_SASL_MECHANISM
  value: "AWS_MSK_IAM"
- name: KAFKA_PROPERTIES_SASL_JAAS_CONFIG
  value: "software.amazon.msk.auth.iam.IAMLoginModule required;"
- name: KAFKA_PROPERTIES_SASL_LOGIN_CALLBACK_HANDLER_CLASS
  value: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

but it fails to authenticate:

08:53:32 [application-akka.actor.default-dispatcher-7] INFO  o.a.k.c.producer.ProducerConfig - ProducerConfig values: 
    acks = 1
    batch.size = 16384
    bootstrap.servers = [XXXXXXXXXXXXXXXXXXXXX:9098, YYYYYYYYYYYYYYYYYYYYYYYY:9098]
    buffer.memory = 33554432
    client.dns.lookup = default
    client.id = datahub-frontend
    compression.type = none
    <http://connections.max.idle.ms|connections.max.idle.ms> = 540000
    <http://delivery.timeout.ms|delivery.timeout.ms> = 120000
    enable.idempotence = false
    interceptor.classes = []
    key.serializer = class org.apache.kafka.common.serialization.StringSerializer
    <http://linger.ms|linger.ms> = 0
    <http://max.block.ms|max.block.ms> = 60000
    max.in.flight.requests.per.connection = 5
    max.request.size = 1048576
    <http://metadata.max.age.ms|metadata.max.age.ms> = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    <http://metrics.sample.window.ms|metrics.sample.window.ms> = 30000
    partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
    receive.buffer.bytes = 32768
    <http://reconnect.backoff.max.ms|reconnect.backoff.max.ms> = 1000
    <http://reconnect.backoff.ms|reconnect.backoff.ms> = 50
    <http://request.timeout.ms|request.timeout.ms> = 30000
    retries = 2147483647
    <http://retry.backoff.ms|retry.backoff.ms> = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = [hidden]
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = AWS_MSK_IAM
    security.protocol = SASL_SSL
    send.buffer.bytes = 131072
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = https
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    <http://transaction.timeout.ms|transaction.timeout.ms> = 60000
    transactional.id = null
    value.serializer = class org.apache.kafka.common.serialization.StringSerializer

08:53:33 [application-akka.actor.default-dispatcher-7] INFO  o.a.k.c.s.a.AbstractLogin - Successfully logged in.
08:53:33 [application-akka.actor.default-dispatcher-7] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.3.0
08:53:33 [application-akka.actor.default-dispatcher-7] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId: fc1aaa116b661c8a
08:53:33 [application-akka.actor.default-dispatcher-7] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1669625613221
08:53:33 [kafka-producer-network-thread | datahub-frontend] INFO  o.a.kafka.common.network.Selector - [Producer clientId=datahub-frontend] Failed authentication with XXXXXXXXXXXXXXXXX (An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.)
08:53:33 [kafka-producer-network-thread | datahub-frontend] ERROR o.apache.kafka.clients.NetworkClient - [Producer clientId=datahub-frontend] Connection to node -2 (XXXXXXXXXXXXXXXXX:9098) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
08:53:33 [kafka-producer-network-thread | datahub-frontend] INFO  o.a.kafka.common.network.Selector - [Producer clientId=datahub-frontend] Failed authentication with YYYYYYYYYYYYYYYYYY (An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.)
08:53:33 [kafka-producer-network-thread | datahub-frontend] ERROR o.apache.kafka.clients.NetworkClient - [Producer clientId=datahub-frontend] Connection to node -1 (YYYYYYYYYYYYYYYYYY:9098) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
...

Any idea what I can do here? Thanks!

I know we do use MSK internally with the frontend.. let me check what our setup looks like

The PR to add the jar has been merged, however it is not yet in a release yet.

Hi @brainy-tent-14503, that's great! Could you please share the link to that PR? I did not find it.

Looking back the PR was for the kafka-setup job here. Likely a modification of this for frontend would work to add the required jar.