Hey guys, Is there a way to connect Pinot to a secure Kafka cluster using Jaas configuration?

@User might have some insights here.

Hi Satyam, yes. I remember @User was settings up something similar some time ago. If you look up his posts via search you’ll find a number of actual configs

@User can you please help find it?

@User ^^

non SSL SASL/JAAS config here - https://dev.startree.ai/docs/pinot/recipes/kafka-sasl

Thanks @User Will this work on the open source release of pinot?

That's what I've been using, so yup!

Getting this error in the controller logs @User

Kafka version: 2.8.1
Kafka commitId: 839b886f9b732b15
Kafka startTimeMs: 1649834197621
[Consumer clientId=consumer-null-648, groupId=null] Subscribed to partition(s): test-topic--2147483648
Metrics scheduler closed
Closing reporter org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for consumer-null-648 unregistered
Transient Exception: Could not get partition count for topic test-topic
org.apache.pinot.spi.stream.TransientConsumerException: org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
	at org.apache.pinot.plugin.stream.kafka20.KafkaStreamMetadataProvider.fetchPartitionCount(KafkaStreamMetadataProvider.java:50) ~[pinot-kafka-2.0-0.10.0-SNAPSHOT-shaded.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.spi.stream.StreamMetadataProvider.computePartitionGroupMetadata(StreamMetadataProvider.java:66) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:69) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.spi.stream.PartitionGroupMetadataFetcher.call(PartitionGroupMetadataFetcher.java:31) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.controller.helix.core.PinotTableIdealStateBuilder.getPartitionGroupMetadataList(PinotTableIdealStateBuilder.java:166) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]

that error is when you try to create the table?

yes

hmmm interesting. In the example that I shared with you I had a Confluent Cloud Kafka cluster connected to a local Pinot

yeah i did the same…

I remember seeing this before though

In this case, lots of messages are there.

ok, so not that

Is this a connectivity issue?

ok, hopefully Kartik can see why it's not working. If that doesn't work I could always share my config with you to help narrow down the problem

Thanks Mark. Working with Satyam to resolve both the JaaS and Spark ingestion issue.