Hello all. We are using Apache Flink jobs with checkpoints enabled on Kubernetes cluster. We are seeing from time to time the following error:

/flink/ha/default/completedCheckpoint6180310b69c1 (No such file or directory)
and 
Could not retrieve checkpoint 2728 from state handle under checkpointID-0000000000000002728. This indicates that the retrieved state handle is broken. Try cleaning the state handle store.

also in the stack:

Failed to initialize high-availability completed checkpoint store
eventually:
JobMaster for job aa524fd45d98dc7bb0b1be7798a87415 failed.
and Fatal error occurred in the cluster entrypoint.

the entire flink pod is killed and restarted but same goes on again and again only clearing the config-maps help. how can we tell flink not to fail in case of job recovery failure we have a mechanism for recreating the job through our application.