Hi there, any help please? I’m struggling to connect external Kafka to Pinot. I have Kafka deployed in one Kubernetes cluster and Pinot in separated one. I’m 100% sure that the communication between these two clusters are correct.
Deployment of Pinot is done by this tutorial
https://docs.pinot.apache.org/basics/getting-started/kubernetes-quickstart
I created Kafka topics, Pinot table and schema according this
https://docs.pinot.apache.org/basics/getting-started/pushing-your-streaming-data-to-pinot
and just changed config pointing to our Kafka brokers
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.topic.name": "transcript-topic",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "our_kafka_url:9092",
"realtime.segment.flush.threshold.size": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.desired.size": "50M",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest"
}
}
But I still do not see any data in my tables.
I’m using Kafka version 2.6.2
Or does Pinot work just with Kafka deployed together with it with usage of the same Zookeeper? I tried to set stream.kafka.zk.broker.url to our Zookeeper but still without success.
Thanks a lot