Hi Everyone. I have a small stack running on localhost integrated with Kakfa. Next I want to integrate Pinot with our production Confluent Kafka platform. The data is mostly in AVRO format AND we have customized Confluent Kafka to require an OKTA auth token. It appears that out of the box is support for JSON payloads from Kafka and HTTP Basic Auth. I have searched for OKTA on this channel and found no hits. Assuming I wanted to make some changes to support AVRO payloads and OKTA auth when integrating with Kafka, can anyone point me at the classes that I should take a look at or study. Do you have any other advice that I ought to know about before I commence down this path? Thanks appreciate it.

We do support SASL authentication support. Is it an option to get API key & secret from Confluent Kafka platform? https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#use-kafka-partition-low-[…]l-consumer-with-sasl_ssl

Avro is already supported

Thanks for the link. For the OKTA token I assume the starting point is the KafkaConnectionHandler class. Do I have that right?

@Steven Hall So, we use Kafka consumer under the hood. We should support as long as the kafka client supports this. I found one documentation on using

OAuth2

authentication for accessing Kafka Cluster using Okta. Can you try out this? you can add the following within streamConfig

security.protocol=SASL_PLAINTEXT
sasl.mechanism=OAUTHBEARER
sasl.login.callback.handler.class=com.oauth2.security.oauthbearer.OAuthAuthenticateLoginCallbackHandler
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required OAUTH_LOGIN_SERVER=<OAuth-server-url> OAUTH_LOGIN_ENDPOINT='/oauth2/default/v1/token' OAUTH_LOGIN_GRANT_TYPE=client_credentials OAUTH_LOGIN_SCOPE=kafka OAUTH_AUTHORIZATION='Basic <encoded-producer-clientId:clientsecret>' OAUTH_INTROSPECT_SERVER=<OAuth-server-url> OAUTH_INTROSPECT_ENDPOINT='/oauth2/default/v1/introspect' OAUTH_INTROSPECT_AUTHORIZATION='Basic <encoded-producer-clientId:clientsecret>';

https://medium.com/egen/how-to-configure-oauth2-authentication-for-apache-kafka-cluster-using-okta-8c60d4a85b43

Thank you. I will try this and report back. It’s going to take just a bit as I am still working through some authn issues on a simple test case to send and read back some messages for our Kakfa environment using the Confluent Kafka Producer and Confluent Kafka Consumers. One we have that working I will pivot to getting it working with Pinot and I will let you know.

please let me know once you get to work on pinot side. we are also interested if this would work 🙂

Will do Seunghyun

Hello, I have also tried to integrate Pinot with Kafka using OAuth authentication with the following configuration:

"sasl.mechanism": "OAUTHBEARER"

_"security.protocol": "SASL_PLAINTEXT"_

_"sasl.jaas.config": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required oauth.client.id=\"<client_id>\" oauth.client.secret=\"<client_secret>\" oauth.token.endpoint.uri=\"<oauth_token_endpoint_uri>";"_

"sasl.login.callback.handler.class": "com.oauth2.security.oauthbearer.OAuthAuthenticateLoginCallbackHandler"

But, I'm getting an exception:

<http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.kafka.common.config.ConfigException: Invalid value com.oauth2.security.oauthbearer.OAuthAuthenticateLoginCallbackHandler for configuration sasl.login.callback.handler.class: Class com.oauth2.security.oauthbearer.OAuthAuthenticateLoginCallbackHandler could not be found.

Do you/the team have any suggestions? Thanks!

@Atul Patil This is a useful tool to look at the contents of your jar and even the package names of the classes in the jar: https://java-decompiler.github.io/ You will want to use this tool to understand what’s in the jar. Pinot shades some of the dependent jars to avoid namespace collision. The Maven shade plugin is used during the dist build to rename various packages. I notice that your error message indicates the namespace org.apache.pinot.shaded.org.apahe.kafka.common.config the Pinot build adds that shade prefix: org.apache.pinot.shaded into some of the classes — that’s right the shade plugin changes the bytecode of some classes to rename the packages. I have a config using Strimzi and have it shaded “stream.kafka.bearer.auth.credentials.source”: “org.apache.pinot.shaded.io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler” > If the class OAuthBearerLoginModule is shaded then you won’t find it as defined in your configuration: >> “sasl.jaas.config”: “org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule > Rather, you may have to define it as “sasl.jaas.config”: “org.apache.pinot.shaded.org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule … > > The jar decompiler will be helpful to look under the hood to see what’s really going on. Alternately, there could be an easier way. Some of those classes might be shaded so that the same dependency jars can be put into the plugins directory where they will be found on the classpath without the shaded names and will not conflict with any of the packages that Pinot includes for things like running unit tests against Kafka. If that idea is right, then your config is fine, but the jars for those dependencies need to be added to the plugin directory. Let me know if this idea works. If it does I may be able to make my future life a lot easier than what I have been doing.