Flink restored from empty state Hi everyone! We're running Flink 1.13 in Amazon Managed Apache Flink. We have managed snapshots (Flink savepoints) enabled and we're using

RESTORE_FROM_LATEST_SNAPSHOT

when updating our ETL job. We don't have any checkpoints enabled at the moment or any scheduled snapshotting. Today we made a change where we removed one operator from a job. The first deployment failed, probably because we did not have AllowNonRestoredState enabled. So we tried again, this time with AllowNonRestoredState enabled. Flink started fine, but for some reason our operators seem to have lost state. For example, an operator counting daily events started counting again from zero. This operator was not changed in the update. Questions to gurus: • Is it possible to see somehow in Amazon Managed Apache Flink which operators start from which state? • Do you have any idea why we lost state? Could it be related to the failure in the first deployment? • Could we have avoided this scenario somehow by using scheduled snapshots or checkpointing? I very much appreciate any comments! And sorry for the n00b questions, I'm just starting with Flink as our team inherited the application from an engineer who left the company 😬

Did you set the UUID of each operator? That’s the tag that Flink uses to match state from snapshot to operator when restored. It could be that the UUID has now changed after you edited the job graph

Hmm good question! All operators using the DataStream API have

uid

set, but the operator in question uses Flink Table API 🤔

Hmm, I think the Flink Table API has not very good guarantees on state compatibility on job graph changes

It all depends on the change that's made in the query if it's indeed compatible or not. It's documented at https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/upgrading/#table-api--sql

It all depends on the change that's made in the query if it's indeed compatible or not.

Thanks for the comments! We did not make any changes to this query in question. Our pipeline has a single event source reading and parsing events from Kinesis. These events are then forwarded to six branches using five side outputs. One of the side outputs has this SQL query in question. One another side output had another query that we removed. The first one failed to restore state. Could these kinds of changes elsewhere in the job graph also give problems in state compatibility?

Yes, because when you use SQL/Table, the job graph gets generator for you. When you change any SQL/Table API in your app, the generated job graph can be changed