Hi Druid community! I use Druid to ingest kafka messages from a topic. I use the protobuf parser to parses messages. However, now we have some messages that are sent with the JSON format (those are watermark messages). I therefore get ParseException and the index_kafka task fails. Does anybody have any idea how I could deal with this issue? Thanks!

normally i would expect the unparseable messages are skipped unless

maxParseExceptions

is set to zero or something

Hello Gian. Thanks for your answer. Yes I tried to either omit this parameter or to put it at max in my supervisor definition:

"tuningConfig" : {

"type" : "kafka",

"maxRowsInMemory" : 100000,

"maxRowsPerSegment" : 5000000,

"maxParseExceptions" : 2147483647,

"basePersistDirectory" : "/opt/druid/data/basePersist"

}

but still get my index_kafka tasks failing with :

org.apache.druid.java.util.common.parsers.ParseException: Protobuf message could not be parsed at c...

Any idea what I could try 🤔?

what version are you using? this might have been fixed in 26 with https://github.com/apache/druid/pull/14183

Thanks! Ah yes, I am using 0.20.2. The ticket mentions explicitly Avro, but I guess it is same problem I have with Protobuf?

i think Protobuf could have also been fixed by that patch

Ok cool, thanks!