Hi, we’ve been trying to setup Apache Pinot ingest...
# troubleshooting
s
Hi, we’ve been trying to setup Apache Pinot ingestion from Azure EventHub (Azure EventHub supports Kafka protocol so that Kafka clients can read from Azure EventHub). All the events within Azure EventHub are Protobuf events. Any ideas on how to make protobuf deserialization work here? The table creation is successful but data is not being ingested.
This is the configuration, if it helps:
Copy code
"streamConfigs": {
          "streamType": "kafka",
          "stream.kafka.consumer.type": "highLevel",
          "stream.kafka.topic.name": "<topic-name>",
          "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufRecordReader",
          "stream.kafka.group.id": "$Default",
          "stream.kafka.client.id": "pinot-test",
          "stream.kafka.security.protocol": "SASL_SSL",
          "stream.kafka.sasl.mechanism": "PLAIN",
          "stream.kafka.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=${password}",
          "stream.kafka.zk.broker.url": "<eventhub-namespace>.<http://servicebus.windows.net:9093|servicebus.windows.net:9093>",
          "realtime.segment.flush.threshold.rows": "0",
          "realtime.segment.flush.threshold.time": "24h",
          "realtime.segment.flush.segment.size": "100M",
          "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
          "stream.kafka.broker.list": "<eventhub-namespace>.<http://servicebus.windows.net:9093|servicebus.windows.net:9093>",
          "realtime.segment.flush.threshold.size": "500",
          "stream.kafka.consumer.prop.auto.offset.reset": "earliest"
        }
k
@Kartik Khare is protobuf supported in real-time?
k
No, Only Avro and JSON are supported in realtime. I can take this up.
s
Okay. Thanks for confirming.