MohammadReza Shahmorady
05/31/2025, 2:28 PMAndreas Bube
06/02/2025, 6:18 AMQuantityPriceAggregatorFunction operation, and any other stateful operations that use the type in question. That would effectively mean those stateful operators start from zero and all state for them is lost. So be careful, and ensure that you really don't need that state!
Kryo serialization doesn't allow state schema evolution. If you want to be able to add and remove fields you need to use another data type, e.g. POJO or Avro, or use custom state serialization.
I have never myself needed to evolve a Kryo serialized type, so I have very little guidance to offer on how to retain the state and evolve to Kryo serialized type. Perhaps the docs on custom serialization helps?
We have disabled Kryo serialization entirely by default in all our Flink jobs, to avoid these kinds of issues. This forces the engineers to do some more work in order to ensure all types are serializable using the POJO serializer, but it keeps us out of these kinds of issues. How get migrate from Kryo to POJO is probably not simple, and I cannot provide any guidance on that either (I imagine it would be painful if you want to retain the state)Pedro Mázala
06/02/2025, 8:48 AMMohammadReza Shahmorady
06/02/2025, 9:01 AMenv.getConfig().registerPojoType(Quantity.class);
env.getConfig().registerPojoType(Product.class);
env.getConfig().registerPojoType(CatalogItem.class);
env.getConfig().registerPojoType(Price.class);
env.getConfig().disableForceKryo();
I added these lines of code.
Is this the correct approach?
And if I want to use Avro, is it enough to just add
env.getConfig().enableForceAvro();
or do I need to write a custom serializer as well?Andreas Bube
06/02/2025, 11:11 AMConfiguration config = new Configuration();
config.set(PipelineOptions.GENERIC_TYPES, false); // env.getConfig().disableGenericTypes(); could also be used, but it's deprecated in favor of the GENERIC_TYPES setting
config.set(PipelineOptions.AUTO_GENERATE_UIDS, false);
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config)
The GENERIC_TYPES = false setting disables Kryo entirely and will make Flink throw an UnsupportedOperationException whenever it encounters a data type that would go through Kryo for serialization. As @Pedro Mázala mentions disabling auto generated UIDs is also good practice. From the docs:
If you do not specify the IDs manually they will be generated automatically. You can automatically restore from the savepoint as long as these IDs do not change. The generated IDs depend on the structure of your program and are sensitive to program changes. Therefore, it is highly recommended assigning these IDs manually.The effectively means that the state will become un-restorable after certain program changes (because the auto-assigned UIDs have changed). So always manually assign UIDs
MohammadReza Shahmorady
06/02/2025, 11:59 AMAndreas Bube
06/02/2025, 12:02 PMMohammadReza Shahmorady
06/02/2025, 12:58 PMAndreas Bube
06/02/2025, 5:18 PM