Vikas Basavaraj Mallapura
12/30/2022, 12:28 AM// --------- initial run --------
RecoverableWriter writer = fileSystem.createRecoverableWriter();
RecoverableFsDataOutputStream out = writer.open(path);
out.write(...);
// persist intermediate state
ResumeRecoverable intermediateState = out.persist();
storeInCheckpoint(intermediateState);
// --------- recovery --------
ResumeRecoverable lastCheckpointState = ...; // get state from checkpoint
RecoverableWriter writer = fileSystem.createRecoverableWriter();
RecoverableFsDataOutputStream out = writer.recover(lastCheckpointState);
out.write(...); // append more data
out.closeForCommit().commit(); // close stream and publish all the data
I was wondering in which scenario storeInCheckpoint method is called ? If it needs to take a checkpoint, why wouldn't it call commit() directly on the output stream ?