Hi there, any help please? I’m struggling to conne...
# troubleshooting
j
Hi there, any help please? I’m struggling to connect external Kafka to Pinot. I have Kafka deployed in one Kubernetes cluster and Pinot in separated one. I’m 100% sure that the communication between these two clusters are correct. Deployment of Pinot is done by this tutorial https://docs.pinot.apache.org/basics/getting-started/kubernetes-quickstart I created Kafka topics, Pinot table and schema according this https://docs.pinot.apache.org/basics/getting-started/pushing-your-streaming-data-to-pinot and just changed config pointing to our Kafka brokers "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.topic.name": "transcript-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.broker.list": "our_kafka_url:9092", "realtime.segment.flush.threshold.size": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.desired.size": "50M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } } But I still do not see any data in my tables. I’m using Kafka version 2.6.2 Or does Pinot work just with Kafka deployed together with it with usage of the same Zookeeper? I tried to set stream.kafka.zk.broker.url to our Zookeeper but still without success. Thanks a lot
m
No, Pinot does not need Kafka to be on same ZK. What do you see in the server logs?