####Urgent help required###
During checkpointing, ...
# troubleshootingc
chunilal kukreja
04/14/2023, 10:00 AM####Urgent help required###
During checkpointing, if the folder where snapshot is to be saved is already present. Like in my case “chk-1” is the folder where snapshot is to be saved is already present. I get below exception & post that job gets restarted.
Copy code
WARN org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 1 for job 000000006e6b13320000000000000000. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Failure to finalize checkpoint.
Caused by: java.io.IOException: Target file file:/opt/flink/pm/checkpoint/000000006e6b13320000000000000000/chk-1/_metadata already exists.
at org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream.getOutputStreamWrapper(FsCheckpointMetadataOutputStream.java:168)m
Martijn Visser
04/14/2023, 10:01 AMWhy should it skip? Checkpointing is an internal mechanism, you shouldn’t influence it
c
chunilal kukreja
04/14/2023, 10:03 AMby skip I meant either use another folder name or overwrite the contents.. But atleast do something, rather it throws an exception & job gets restarted
m
Martijn Visser
04/14/2023, 10:04 AMI don’t understand how you ended up in this situation
Martijn Visser
04/14/2023, 10:04 AMDid you have a failure and tried to restart?
Martijn Visser
04/14/2023, 10:05 AMAnd what Flink version are you using?
c
chunilal kukreja
04/14/2023, 10:06 AMNo failure happened
1.16.0 version
chunilal kukreja
04/14/2023, 10:07 AMwhen you start the flink job using savepoint/checkpoint flag, if the checkpoint folder already exists this exception occurs & job gets restarted
chunilal kukreja
04/14/2023, 10:08 AMif you see the casued by statement;
“Target file file:/opt/flink/pm/checkpoint/000000006e6b13320000000000000000/chk-1/_metadata already exists.”
here the folder “chk-1" already exists hence the exception
n
Neha
04/14/2023, 11:09 AMI am using 1.16.1 and have tested starting with savepoint and checkpoint, no issues observed
m
Martijn Visser
04/14/2023, 12:43 PMwhen you start the flink job using savepoint/checkpoint flagWhat flag do you mean?
Martijn Visser
04/14/2023, 12:45 PMThere isn't enough information to answer this question:
1. When did the error occur, is this when after you've starting a job from an existing check/savepoint, or is this for a new job that you started?
2. If you did restart, how did you do that? Did you use https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/#starting-a-job-from-a-savepoint or something else?
c
chunilal kukreja
04/19/2023, 12:29 PMAnswers:
a) Error occurs after starting job with an existing check/savepoint.
b) We have k8s cluster where we have deployed flink cluster. When exception occurs, job gets restarted and in our code we have defined restart strategy (3 counts). If 3 times job gets restarted, post that k8s pod gets restarted.
m
Martijn Visser
04/19/2023, 1:47 PMa) Error occurs after starting job with an existing check/savepoint.Are you sure that the account you're using to write snapshots to filesystem is also allowed to change and remove objects?
c
chunilal kukreja
04/20/2023, 2:27 PMyeah, user has proper rights assigned.
chunilal kukreja
04/27/2023, 2:28 PMhi Martijn,
Let me know if you need any additional details.