Hi all, Is there a way to ignore Avro (Confluent Schema Registry) deserialization errors using the Flink SQL API? For JSON, CSV and Protobuf there is a configuration suffix

*.ignore-parse-errors

, but it is not available for Avro.

Are you talking about AVRO or Confluent AVRO? For the last one, that option shouldn’t exist because then schema registry has failed 😅

Confluent Avro. I’m getting this:

Caused by: java.io.IOException: Failed to deserialize consumer record ConsumerRecord(topic = [...], partition = 1, leaderEpoch = 118, offset = 274419, CreateTime = 1686247521172, serialized key size = 2, serialized value size = 124, headers = RecordHeaders(headers = [], isReadOnly = false), key = [B@1a2d3710, value = [B@587f5aed).

Then most likely you have misconfigured your table

Yes, but could not find anything about error handling

That’s because schema registry prevents writing records that are incorrectly serialized

As far as I know, the Kafka broker does not enforce a schema on the message key and value, it just sees bytes. The producers and consumers just use schema registry by mutual agreement. This job was working before. My guess is that some arbitrary data was written to the topic recently. It is in a lower environment, so it is ok, but I would like to just ignore it

That’s true

I agree with Martijn here. As long as you use schema registry there should not be any issue with the parsing records and there is no need of

ignore-parse-errors

. So either there is configuration issue or some producers pushed the message without schema registry (say a simple console push from your local). The simple option is to use the simple hello world consumer from that message offset and then decode using the same schema or debug from there. In any case, if you want to retain the Flink state, as there is no other option for you to skip, you can try any of the following, 1. Patch something like this - this is for plain Avro, for confluent it should be similar under ConfluentRegistryAvroDeserializationSchema 2. Just simply delete the message from Kafka like this using console scripts (Section Option 2: Record Deletion) If you don't want the Flink stat, then just do a fresh start again just right after the faulty message.

Yes, I found a single json message inside this topic. I’ve manually deleted it. I will try to create this patch either way. Thank you!