If I have a cancelled / terminated job for a cluster with HA enabled, what’s the easiest way to determine the location of the last retained checkpoint?

Go to the checkpoint directory and find the latest file

I also have s3 entropy enabled, would that make it more difficult finding the latest file?

I have no idea

Maybe we could improve this in the upstream Flink project. For example, store the latest retained checkpoint path in the JRS(job result store) along with job status.

That would be a great improvement for the operator functionality as well

cc @Matthias Maybe you are also interested.

Yes, I'm interested in getting this feature out. I'm happy to create a FLIP after I'm done with the leader election work (FLIP-285). But I would be also willing to help if someone else decides to pick it up earlier. 🙂

Maybe creating a dedicated FLIP for adding checkpoint location in JRS is an overkill.🤔

Yea, this is a relatively minor change with no side effects. I think no jira is necessary

might be. I haven't thought this through entirely. The idea is to merge the

ExecutionGraphInfoStore

and the

JobResultStore

into a single component/interface. IIRC, this might expose other meta information besides the

JobResult

in the latest checkpoint. This information (not only the checkpoint) might become useful to external projects. We might want to impose stricter policies on the data structure (making it @Public/@PublicEvolving API). That's why, I was thinking of a FLIP, initially.

Make sense. A stable API for external project is really important.