I m pushing avro serialized data in binary format in kafka a Apache Pinot #troubleshooting

I’m pushing avro serialized data in binary format ...

Romil Punetha

08/06/2022, 6:54 AM

I’m pushing avro serialized data in binary format in kafka and want to create a realtime table on top of it. However, I’m getting the following error when using `org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder`:

Copy code

Caused by: java.lang.IllegalStateException: Avro schema must be provided

When using

org.apache.pinot.plugin.inputformat.avro.KafkaAvroMessageDecoder

I get:

Copy code

Caused by: java.lang.NullPointerException
at org.apache.pinot.plugin.inputformat.avro.KafkaAvroMessageDecoder.parseSchemaRegistryUrls(KafkaAvroMessageDecoder.java:303)

How should I provide avro schema urls? I can’t create an avro schema from UI. What else?

Romil Punetha

08/06/2022, 7:55 AM

I found how to supply the URls here:

<https://www.mail-archive.com/dev@pinot.apache.org/msg02740.html>

However, I can see that the code appends

latest_with_type

which does not work with apicurio registry. Any workarounds?

Romil Punetha

08/06/2022, 12:14 PM

I fixed the issue with SimpleAvroMessageDecoder. Had to mention the schema. Now I’m getting this:

Copy code

java.lang.ArrayIndexOutOfBoundsException: Index 17 out of bounds for length 2
	at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:460) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:284) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder.decode(SimpleAvroMessageDecoder.java:97) ~[pinot-avro-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder.decode(SimpleAvroMessageDecoder.java:44) ~[pinot-avro-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:552) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:430) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:623) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-1cb172aae7f1d9cd5855c74b9f004405aeaedeaa]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Romil Punetha

08/07/2022, 8:06 AM

Tried using

KafkaConfluentSchemaRegistryAvroMessageDecoder

Used `quarkus-confluent-registry-avro`for serialization Getting :

Copy code

org.apache.pinot.shaded.org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.pinot.shaded.org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

Open in Slack

Previous Next