Hi Team, I am encountered with an issue regarding ...
# troubleshooting
a
Hi Team, I am encountered with an issue regarding flink state restoration, Actually what is happening here is like initially there is Flink job with ProcessFunction, AsyncRichFunction and KeyedProcessFunction. And just mentioned the uid (using .uid()) for KeyedProcessFunction only (i.e. for only one operator) and I'm using state in this function, then I added one more KeyedProcessFunction and using state here also and assign uid for each operators and keeping initial uid same for that KeyedProcessFunction just added name (using .name()) for that. But whenever I'm trying to deploy (upgrade) the Job with this changes the task managers are not spawning up and job is not able to restore the state even though uid will remain same for that operator before and after the change. Here I'm using RocksDB as state backend with efs. I tried to retrieve the state from savepoint also using initailSavepointPath and upgradeMode as savepoint But here also getting same error like not able to restore the state. can anyone help here? Thank you.
d
Probably due to the fact that Flink uses a combination Operator name and UID to uniquely identify stateful operators across job upgrades or savepoint restores
If two different operators claim the same UID that could be a problem
Ensure unique UID for each new stateful operator no exceptions. Don’t rely just one name change.
Instead migrate state if you need to from old to new operator. Do not keep same UID
Be sure to to use correct upgradeMode when doing upgrades if you want to resume from a particular save point
Might need to do a manual savepoint & restart
Make sure all task managers have access to the EFS and that paths are consistent across redeployments
Always do the simplest thing that works. Before doing an upgrade try just creating a new job with the new topology ensuring all UID are correctly set and you can do save state and restore without error. Do this first then the upgrade.
a
No but as per Flink Documentation Flink used UID as identifier for the operator right, name is just for displaying/naming purpose. I'm trying to restore the data for that older operator only form the state, so to identify that operator UID should be the same right? I already take care that each operator should have unique UID. If I changes the UID for older one in new changes, I might loose the data in state right which I don't want
d
I did not say to change it for older ones. But new ones should get their own unique UID
Just make sure that you are using unique UID for ALL new operators. that’s for sure.
a
yes I already considered this and handled such that each operator should unique UID but still I'm not able to restore the state
d
Well you might need to create a job to migrate state from old operator to the new one.
a
then how can I restore that state into the current job which has new changes?