Hi trying to understand state migration and `compatibleAfter Apache Flink #troubleshooting

Hi, trying to understand state migration and `comp...

David Causse

07/10/2026, 2:07 PM

Hi, trying to understand state migration and

compatibleAfterMigration

. Initially my understanding was that the old serializer is used to read the stored bytes, the instance object is then written with the new serializer and these new bytes are read by the new serializer, in short I would have assumed that it ensured that the instances received in the operators are always constructed using the new serializer. But I'm hitting a bug where I suspect that the instance read by the old serializer is simply passed to the operator without any migration. (Note this is the operator state of the Async operator). Could someone clarify my understanding about state migration with

compatibleAfterMigration

? Thanks! 🙏

David Causse

07/15/2026, 8:46 AM

While digging into flink runtime, I discovered that

COMPATIBLE_AFTER_MIGRATION

is only taken into account with rocksdb/ForStStateBackend keyed state backends. I suspect that the reason is that with these backends the state is not guaranteed to be fully rewritten on every checkpoints/savepoints and that an explicit migration step is mandatory to avoid keeping very old versions in the state. For operator states or the HashMapState backend I suspect that the migration will actually happen on the next checkpoint/savepoint. A migration might also occur when pushing the record to downstream operators (via

CopyingChainingOutput.pushToOperator

) but in this case flink appears to use

Serializer::copy

and this is actually where our custom serializer failed, it did not expect to have to copy an instance obtained from an old version of the serializer. In short my understanding is that

COMPATIBLE_AFTER_MIGRATION

behaves similarly to

COMPATIBLE_AS_IS

for: • operator state • HashMapState keyed states

Open in Slack

Previous Next