Kasper Hermansen
08/15/2023, 10:46 AM2023-08-15 08:29:34,724 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: read lw-go-events from kafka -> Process -> Sink: write to structured datalake (4/4)#2297 (0372d9bdecc05d337c0192d7b7480ac0) switched from RUNNING to FAILED.
java.io.IOException: Recovering commit failed for object source=lw-go-events/topic=kyc.BirthCountryUpdated/day=2023-08-14/part-3-14035881. Object does not exist and MultiPart Upload rNqwyLpFFWSrt5uT3xmgvNbjRgSK9zDgMnta7W3Q7ufXo9pHey_O0bm08ue90cw8X4tobxDCmPOYiNN2EadGBD7.IUwVBBYvds2na7QxMDEQ9b7ENCYmUop5d8pZ5h7cxBCeZvn7xgySDPEEBIs7qA-- is not valid.
at org.apache.flink.fs.s3.common.writer.S3Committer.commitAfterRecovery(S3Committer.java:123) ~[?:?]
at org.apache.flink.streaming.api.functions.sink.filesystem.OutputStreamBasedPartFileWriter$OutputStreamBasedPendingFile.commitAfterRecovery(OutputStreamBasedPartFileWriter.java:218) ~[flink-table-blink_2.12-1.12.1.jar:1.12.1]
I’ve only got a checkpoint later than this issue, so a recovery doesn’t work. And no save point stored as well.
Is there a way in flink which I can recover from this issue
we’re currently on flink 1.12