Hey everyone :wave: Last week we had an issue and ...
# troubleshooting
h
Hey everyone 👋 Last week we had an issue and i was wondering if anyone has experienced this before or can shed any light into it. We had TaskManager was terminated by the JobManager (due to not responding, which is fine). But when the new TaskManager started up, it was unable to recover the KeyedState for one of the operators. Thanks in advance for your help.. more info in thread
Setup: • We run Flink (version 1.19.0)in Application mode in Kubernetes using the Apache Flink Operator (version 1.8.0). • Our application is using
pyflink
in streaming mode • We use
savepoint
upgrade mode • RocksDb for statebackend • We have
state.backend.changelog
enabled with
state.backend.changelog.periodic-materialize.interval
set to
10 min
and
filesystem
as storage. • Checkpointing interval is set to
1 min
with a
1 min
min-pause and we retain
20
checkpoints. Issue: • When the new Taskmanager started up, it was unable to recover the state for one of the keyed state operators. • The exception we received is exactly the same as the one mentioned in this Flink Jira issue: FLINK-28440. The only difference is instead of a
FileNotFoundException
we received
NullPointerException
• Once that happens, the job is restarted, and the process repeats itself Solution: • We were able to recover by using manual recovery steps mentioned in Flink documentation. Since we have claim_mode set to
CLAIM
we didn’t have a savepoint to recover from, so we used the checkpoints we have retained. • We tried the latest checkpoint, before this issue started, but that didn’t help, as when the Taskmanager started up, it failed with the same exception • We tried an older one, and the same issue happened • Our third attempt with another older checkpoint resolved the issue, and we’re able to recover the state and the taskmanager started up fine and started processing. It’s the first time we have this issue, a few notes is change log was only enabled around 6 months ago, and we upgraded from Flink 1.17 to 1.19 1 month ago. Give the above Flink JIRA issue, my suspicion is this is caused by enabling the
changelog
experimental feature. Please let me know if any has experienced this issue before. One caveat is we lost the logs of the issue as our retention period has kicked in deleted old logs 😞