Hey team 👋 Is there any way to pull the Avro Schema from the registry again when there is a problem deserializing ? I found several times that the Avro schemas changes and Flink job fails because it is using a Cached version of the Schema Registry. To fix we have to suspend and run again the job to pull the latest version Thanks

What is your setup? As far as I know how that should work, the Avro bytes send (e.g. over the Kafka) contains a version of schema that was used to serialize them, so it should always fetch that specific version to deserialize. And then will ether use latest schema from schema registry or schema that you have compiled in (depending on your setup) to do some migrations. Do you do some incompatible changes and then clear the schema registry?

Thanks @Grzegorz Liter, We are using SQL API. The schemas changes are compatible since they are set to be backward compatible, the problem arises when an enum is changed. For instance you have a topic that has the values of YES as Enum. If you start producing messages with No and the schema changed to be YES, NO, is when Flink starts to fail and we need to suspend and run again.

Ah yeah, enums are super annoying, they are not forward compatible. I am avoiding it in my projects, just use plain string. You are using Kafka or what are you reading from?

Yes correct this are defined on Avro schema already from the producers

i know there is small bug but it happens when the bytes stored in Kafka message are not actual Avro. Then the deserializer gets broken and needs to be recreated. But don't think it applies to your case. IMO for you I would go one of two paths: 1. drop enums usage 2. since enums are not forward compatible you need to always first redeploy consumer downstream

Thanks a lot for the help !

I have very limited experience with SQL, mainl yworking with DataSteram