Hey 👋 , Recently we saw significant slower flink checkpoints caused s3 throttling 1000's MULTI_OBJECT_DELETES requests for deleting directory markers. Throttling was causing significantly slower checkpoints. We changed our marker retention policy to

fs.s3a.directory.marker.retention:keep

to prevent issuing of the

MULTI_OBECT_DELETES

. Does anyone have experience running with

keep

and the associated long term performance characteristics since we will not delete any directory markers? Flink version : 1.16 Checkpoint Remote storage : S3 hadoop plugin

The S3 Hadoop isn’t recommended for checkpointing, I recommend to use the S3 Presto one