Hamza Haider
07/17/2024, 3:03 PMHamza Haider
07/17/2024, 3:03 PMpyflink in streaming mode
• We use savepoint upgrade mode
• RocksDb for statebackend
• We have state.backend.changelog enabled with state.backend.changelog.periodic-materialize.interval set to 10 min and filesystem as storage.
• Checkpointing interval is set to 1 min with a 1 min min-pause and we retain 20 checkpoints.
Issue:
• When the new Taskmanager started up, it was unable to recover the state for one of the keyed state operators.
• The exception we received is exactly the same as the one mentioned in this Flink Jira issue: FLINK-28440. The only difference is instead of a FileNotFoundException we received NullPointerException
• Once that happens, the job is restarted, and the process repeats itself
Solution:
• We were able to recover by using manual recovery steps mentioned in Flink documentation. Since we have claim_mode set to CLAIM we didn’t have a savepoint to recover from, so we used the checkpoints we have retained.
• We tried the latest checkpoint, before this issue started, but that didn’t help, as when the Taskmanager started up, it failed with the same exception
• We tried an older one, and the same issue happened
• Our third attempt with another older checkpoint resolved the issue, and we’re able to recover the state and the taskmanager started up fine and started processing.
It’s the first time we have this issue, a few notes is change log was only enabled around 6 months ago, and we upgraded from Flink 1.17 to 1.19 1 month ago. Give the above Flink JIRA issue, my suspicion is this is caused by enabling the changelog experimental feature.
Please let me know if any has experienced this issue before. One caveat is we lost the logs of the issue as our retention period has kicked in deleted old logs 😞