Hi, I am running into below issue when I point my Pinot Table to a Kafka topic which has data in Avro format. Any pointers on how I can debug/triage on which field is throwing a null pointer. Any help much appreciated. Thanks !

Caught exception while reading message using schema: <redacted>
java.io.EOFException: null
	at org.apache.avro.io.BinaryDecoder$ByteArrayByteSource.readRaw(BinaryDecoder.java:966) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:372) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:289) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:209) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:469) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:459) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder.decode(SimpleAvroMessageDecoder.java:89) [pinot-avro-0.10.0-SNAPSHOT-shaded.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder.decode(SimpleAvroMessageDecoder.java:43) [pinot-avro-0.10.0-SNAPSHOT-shaded.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:507) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:416) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:576) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-078c711d35769be2dc4e4b7e235e06744cf0bba7]
	at java.lang.Thread.run(Thread.java:829) [?:?]

pinot use

_avroRecordToReuse = _datumReader.read(_avroRecordToReuse, _binaryDecoderToReuse);

which is calling DatumReader.read, and if this method call itself doesn’t give corresponding error, then it’s not possible to know it. From the stacktrace, it seems that the message itself is corrupted.

It's hard to understand this stacktrace without more useful message from avro stacktrace. So I am curious what version of avro you are using. Also, you have to walk along your schema to possibly identify which field might be null.

Yes I had an issue with the schema not matching data. So i added one field at a time and figured out the isssue. Thanks for the help