Hey 👋 , Flink version : 1.13 State Backend: rocksdb incremental checkpointing We recently saw some data loss after a checkpoint completed followed by a TM shutdown. The resulting job was unable to recover with the following error. We recovered by rolling back to the latest -1 checkpoint.

INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - deduplicate   (193/360) (ccd789f9a61f9a361926927ae4faa43d) switched from INITIALIZING to FAILED on  (dataPort=6121).
[2023-03-03 09:00:03.121693] org.apache.flink.util.SerializedThrowable: Exception while creating StreamOperatorStateContext.
[2023-03-03 09:00:03.121733] 	at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254) ~[s3_ha_store_deploy.jar:?]
[2023-03-03 09:00:03.121829] 	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272) 
[2023-03-03 09:00:03.121869] 	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:441) 
~~~

[2023-03-03 09:00:03.123298] Can't access /1197217.sst: IO error: while stat a file for size: /pay/flink/tmp/job_e3f516c4bdd0854e35d31b90fa9f3296_op_KeyedProcessOperator_62787344f3050171e45f898702868550__193_360__uuid_f62ff828-56a7-4a12-8abb-670f4ccc65ba/db/1197217.sst: No such file or directory
[2023-03-03 09:00:03.123347] Can't access /1197223.sst: IO error: while stat a file for size: /pay/flink/tmp/job_e3f516c4bdd0854e35d31b90fa9f3296_op_KeyedProcessOperator_62787344f3050171e45f898702868550__193_360__uuid_f62ff828-56a7-4a12-8abb-670f4ccc65ba/db/1197223.sst: No such file or directory
[2023-03-03 09:00:03.123394] Can't access /1197237.sst: IO error: while stat a file for size: /pay/flink/tmp/job_e3f516c4bdd0854e35d31b90fa9f3296_op_KeyedProcessOperator_62787344f3050171e45f898702868550__193_360__uuid_f62ff828-56a7-4a12-8abb-670f4ccc65ba/db/1197237.sst: No such file or directory
[2023-03-03 09:00:03.123441] Can't access /1163418.sst: IO error: while stat a file for size: /pay/flink/tmp/job_e3f516c4bdd0854e35d31b90fa9f3296_op_KeyedProcessOperator_62787344f3050171e45f898702868550__193_360__uuid_f62ff828-56a7-4a12-8abb-670f4ccc65ba/db/1163418.sst: No such file or directory
[2023-03-03 09:00:03.123514] Can't access /1118503.sst: IO error: while stat a file for size: /pay/flink/tmp/job_e3f516c4bdd0854e35d31b90fa9f3296_op_KeyedProcessOperator_62787344f3050171e45f898702868550__193_360__uuid_f62ff828-56a7-4a12-8abb-670f4ccc65ba/db/1118503.sst: No such file or directory

Shutdown hook threw the following error:

[2023-03-03 04:50:12.071855] WARN  og.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager [] - Exception while deleting local state directory for allocation id fa09ca77ce735c8e7383rbe61abbf07.
[2023-03-03 04:50:12.071921] java.nio.file.DirectoryNotEmptyException: /pay/flink/taskmanager-state/localState/aid_fa09ca77ce73995c8e7383be61abbf07/jid_fa98b0d9e39594b63269faf0fe829f/vtx_62787344f3050171e45f8987068550_sti_192/chk_71201
[2023-03-03 04:50:12.071972] 	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242) ~[?:1.8.0_352]
[2023-03-03 04:50:12.072021] 	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) ~[?:1.8.0_352]
[2023-03-03 04:50:12.072073] 	at java.nio.file.Files.deleteIfExists(Files.java:1165) ~[?:1.8.0_352]
[2023-03-03 04:50:12.072110] 	at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:332) ~[?]
[2023-03-03 04:50:12.072167] 	at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:308) ~[?]
[2023-03-03 04:50:12.072223] 	at org.apache.flink.util.FileUtils.guardIfNotThreadSafe(FileUtils.java:383) ~[?]
[2023-03-03 04:50:12.072270] 	at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:247) ~[?]
[2023-03-03 04:50:12.072330] 	at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:361) ~[?]
[2023-03-03 04:50:12.072377] 	at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:322) ~[?]
[2023-03-03 04:50:12.072415] 	at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:308) ~[?]
[2023-03-03 04:50:12.072454] 	at org.apache.flink.util.FileUtils.guardIfNotThreadSafe(FileUtils.java:383) ~[?]
[2023-03-03 04:50:12.072491] 	at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:247) ~[?]
[2023-03-03 04:50:12.072531] 	at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:361) ~[?]
[2023-03-03 04:50:12.072569] 	at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:322) ~[?]
[2023-03-03 04:50:12.072614] 	at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:308) ~[?]
[2023-03-03 04:50:12.072654] 	at org.apache.flink.util.FileUtils.guardIfNotThreadSafe(FileUtils.java:383) ~[?]
[2023-03-03 04:50:12.072703] 	at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:247) ~[?]
[2023-03-03 04:50:12.072752] 	at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:361) ~[?]
[2023-03-03 04:50:12.072798] 	at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:322) ~[?]
[2023-03-03 04:50:12.072842] 	at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:308) ~[?]
[2023-03-03 04:50:12.072889] 	at org.apache.flink.util.FileUtils.guardIfNotThreadSafe(FileUtils.java:383) ~[?]
[2023-03-03 04:50:12.072924] 	at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:247) ~[?]
[2023-03-03 04:50:12.072970] 	at org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager.cleanupAllocationBaseDirs(TaskExecutorLocalStateStoresManager.java:289) ~[?]
[2023-03-03 04:50:12.073012] 	at org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager.shutdown(TaskExecutorLocalStateStoresManager.java:237) ~[?]
[2023-03-03 04:50:12.073049] 	at org.apache.flink.util.ShutdownHookUtil.lambda$addShutdownHook$0(ShutdownHookUtil.java:39) ~[?]
[2023-03-03 04:50:12.073081] 	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_352]