Hi all, anyone knows well the flink checkpoint to s3? Description : currently when upscale parallelism flink job, it could trigger 503 exception with msg like

"please reduce your request rate"

, following aws s3 recommendation, we are supposed to do retry with exponentielle backoff retry. For information, we are using facebook presto library to do s3 operation with protocole

s3p://

after digging into library I think I have found the

upload

method of this library https://github.com/prestodb/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/s3/PrestoS3FileSystem.java#L1302 As you can see we have

upload.waitForCompletion();

which will block the thread until upload finish, it will do retry with exponentielle backoff in background with default aws client strategy https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/retry/PredefinedBackoffStrategies.java Problème : In our case when we have s3 503 errors we expect flink take a little more time to do checkpoint to keep retrying and force S3 to autoscale at the same time, however we see our checkpoint upload being interrupted really fast(14 seconds) in such scenario which match the catch exception in that

upload

method

catch (InterruptedException e) {
                STATS.uploadFailed();
                Thread.currentThread().interrupt();
                throw new InterruptedIOException();
            }

and we see a lot of

InterruptedException

in our logs Question 🙏 : Anybody could tell us if there is some conf on flink side we can change to avoid our checkpoint s3 upload being interrupted ? Or if you have a idea about the process which interrupt the checkpoint in this case? Because this problème is becoming a bottleneck for us to scale out our flink application 🙏