Hi there! I am currently facing some issue with the frontend and the connection to AWS-MSK. I am greeted in the frontend-logs by:

failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.

I configured the helm-chart via

springKafkaConfigurationOverrides

for AWS-MSK and I don’t see similar errors in the backend-logs - everything related to MSK works fine for the backend. So I assume the configuration itself is fine. The environment variables in the frontend-pod/container seem to be also fine. All SASL/MSK related stuff that we configured gets routed to the envs correctly. However, in the frontend-logs i also find

sasl.client.callback.handler.class = null

in the part where the kafka-producer config is logged. All other SASL/MSK related stuff is set correctly in the producer config though. So it seems that the application is not correctly reading / passing to the producer what is stated in the environment variable

KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

. The only interesting thing I noticed compared to the backend is that the relevant kafka-environment variables are prefixed with

SPRING_

whereas the envs in the frontend start with

KAFKA_

. not sure whether that’s an issue / related here. Both frontend and backend are on v0.9.5, helm-chart version is 0.2.119 Thanks in advance for any pointers!

Hey Dominik, Thanks for sharing this with the channel. Will get back to you with this.

@bulky-electrician-72362 after some investigation i found that the frontend

application.conf

uses another/the wrong environment variable:

analytics.kafka.sasl.login.callback.handler.class = ${?KAFKA_PROPERTIES_SASL_LOGIN_CALLBACK_HANDLER_CLASS}

but what’s set in the env is called

KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS

i think we could rename this in

springKafkaConfigurationOverrides

since afaik the helm-chart simply uses the config-key as env-key/name with some prefix but then it probably won’t work in the backend anymore?! Could you maybe confirm / double-check this? Thanks!

hey,

helm-chart simply uses the config-key as env-key/name with some prefix

it's true helm, just prefexies and convert the the values provided in the values.yaml

KAFKA_PROPERTIES_{{ $configName | replace "." "_" | upper }}

so to set

KAFKA_PROPERTIES_SASL_LOGIN_CALLBACK_HANDLER_CLASS

you need to set

sasl.login.callback.handler.class

under

springKafkaConfigurationOverrides

why do you think that?

because it currently works for the backend with

sasl.client.callback.handler.class

. of course we could set both but this had no effect for the frontend as i just tested it. in the frontend logs the producer now has:

sasl.client.callback.handler.class = null
sasl.login.callback.handler.class = class software.amazon.msk.auth.iam.IAMClientCallbackHandler

so it needs

sasl.client.callback.handler.class

instead of

sasl.login.callback.handler.class

however the

application.conf

file only accounts for

analytics.kafka.sasl.login.callback.handler.class = ${?KAFKA_PROPERTIES_SASL_LOGIN_CALLBACK_HANDLER_CLASS}

Looks to me that we instead need

analytics.kafka.sasl.client.callback.handler.class = ${?KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS}

in the

application.conf

?!

this was recently added: https://github.com/datahub-project/datahub/blob/master/datahub-frontend/app/client/KafkaTrackingProducer.java#L96 by @brainy-tent-14503 he might have more context here

thanks a lot 🙏

thanks @bulky-electrician-72362

It's been merged, you can try build your own image, or wait for the next release (next week), anyhow interested in your feedback

@bulky-electrician-72362 thanks! we just deployed one of your development tags and no errors in the frontend-logs anymore 🙌