How to avoid checkpoint failures ...
# troubleshootingp
PRASHANT GUPTA
01/18/2023, 4:36 PMHow to avoid checkpoint failures Context --> I have enabled checkpointing , with below configs checkpointDurationMs: 60000
checkpointTimeoutMs: 180000
maxCheckpointRetained: 10
tolerableFailedCheckpoint: 100 I am using flink kubernetes operator , whenever i uninstall and reinstall the flink deployments i see these exceptions in my jobmanager and my checkpoints fails.
Caused by: java.io.IOException: Target file file:/data/datas-template/checkpoints/00000000067179110000000000000000/chk-1/_metadata already exists.g
Gyula F贸ra
01/18/2023, 8:10 PMseems like the checkpoint path conflicts for your jobs
Gyula F贸ra
01/18/2023, 8:10 PMwhat operator version are you using?
Gyula F贸ra
01/18/2023, 8:12 PMI thought we already fixed this issue in newer operator releases by generating random jobIds everytime the job starts from an empty state
Gyula F贸ra
01/18/2023, 8:12 PMthe manual workaround is to adjust the checkpoint path before you create the deployment initially
p
PRASHANT GUPTA
01/19/2023, 4:44 AMYes , we are seeing same folder on the external azure file storage getting used even with new deployment. Operator version is 1.0.0
PRASHANT GUPTA
01/19/2023, 4:45 AMAdjusting the checkpoint path works , but that would be something we would have to do for every new deployment
s
sap1ens
01/19/2023, 4:52 AMOperator version is 1.0.0That's pretty old though, try upgrading to the latest version.
p
PRASHANT GUPTA
01/19/2023, 5:03 AMYes will upgrade now and give a try
馃憤 1
PRASHANT GUPTA
01/27/2023, 7:27 AMUpdated Flink operator to 1.3.0 and issue is resolved. Now every new deployment get a new Job id and hence a new folder is getting created in target directory for checkpoint storage. Thanks a lot for the inputs
鉂わ笍 1