Hi,
Do you have a stateful streaming Flink job that has big state? Did you ever had to wait hours for your job to start up just because you have added a new field to your state model? I certainly did ans started to ask myself a question: why I don't ever need to wait for all files to be rewritten in Data Lake just because I added new column? Why I am not rewriting all messages in Kafka just because some field was dropped? It is because that Flink hold only one schema at a time so it has to rewrite all state bytes when state schema changes.
My solution: create a serializer that utilizes same approach as Confluent Avro Kafka serializer, that is to register schema in schema registry and store the schema id along with the data. Then when we deserialize bytes we read the data in old schema and project to current runtime schema.
I would like to invite you to test the solution
github.com/gliter/flink-schema-resolving-avro-type-serializer
central.sonatype.com/artifact/io.github.gliter/schema-resolving-avro-serializer/overview
There is even option that allows us to use this new serializer on top of an existing job that already uses Avro serializer.
I have also a plan to embed schemas inside state serializer, so we don't need external schema registry.