Hi Team, We are using flink clusters with Elastic File System on AWS. Due to some cost issue we are planning to migrate the backend from EFS to S3 for obvious reasons. Migration is working fine for our small clusters in lower environments. While migrating from EFS backend to S3 backend on our production environment we are recovering with a state of almost 500Gb which is absolutely working fine with the path from EFS and running a job with new s3 backend. Incremental Checkpointing is also working as per our expectation. Issue is coming when we are trigger a savepoint again on the job after migrating it to s3. Seems like it is not creating a savepoint when the job is recovered from some other state backend. Are we missing anything which needs to be taken care? My configuration and setup is using presto binary to write on s3. While creating a savepoint we are getting below error

2023-09-13 22:19:43,192 ERROR org.apache.flink.client.cli.CliFrontend                      [] - Error while running the command.
org.apache.flink.util.FlinkException: Triggering a savepoint for the job 64b85351ced8c27da0b07fea17a6f0c6 failed.
        at org.apache.flink.client.cli.CliFrontend.triggerSavepoint(CliFrontend.java:801) ~[flink-dist-1.15.1.jar:1.15.1]
        at org.apache.flink.client.cli.CliFrontend.lambda$savepoint$9(CliFrontend.java:771) ~[flink-dist-1.15.1.jar:1.15.1]
        at org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1026) ~[flink-dist-1.15.1.jar:1.15.1]
        at org.apache.flink.client.cli.CliFrontend.savepoint(CliFrontend.java:767) ~[flink-dist-1.15.1.jar:1.15.1]
        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1096) ~[flink-dist-1.15.1.jar:1.15.1]
        at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1156) ~[flink-dist-1.15.1.jar:1.15.1]
        at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) [flink-dist-1.15.1.jar:1.15.1]
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1156) [flink-dist-1.15.1.jar:1.15.1]
Caused by: org.apache.flink.util.SerializedThrowable: org.apache.flink.runtime.checkpoint.CheckpointException: Asynchronous task checkpoint failed.
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
        at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:252) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
        at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1387) ~[flink-dist-1.15.1.jar:1.15.1]
        at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93) ~[?:?]
        at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[?:?]
        at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92) ~[?:?]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
        at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45) ~[?:?]

This sounds strange indeed. I assume you migrated by copying a savepoint into S3, and starting a new job from it? If anything, savepoints have a more universal format, and it is very peculiar that incremental checkpoints work but savepoints don’t. Did you try to trigger a full checkpoint instead of the savepoint?