Hello everyone! I am currently migrating a Flink application (Java) from EMR to AWS EKS. We currently save checkpoints using RocksDB in HDFS. I am wondering if somebody has a recommendation or could point me to some doc in which I could see options and best practices to setup checkpoint store in EKS or K8S in general? We previously used S3 for checkpoint but the context I get is that there was some throttling and the team decided to move to RocksDB (not much info in that front unfortunately). Any help is appreciated! Thanks in advance!

We use the RocksDB state backend and store savepoint and checkpoints in S3. We use the Presto (

s3p://

) file system plugin and incremental checkpoints. I haven't heard any teams complain about checkpoints or savepoint getting "throttled" by S3. But TBH we're not running massive jobs. State for the larger jobs sit around 30GB, so we're definitely not pushing any limits here. For us RocksDB + S3 just seems to work fine 🤷

Curious, why did you decide to move from EMR? Because we are actually considering EMR

According to the team's analysis, some reasons such as multi-AZ, better maintenance (cluster and flink versions upgrade), cost (reduction possibly using spot instances), Flink K8s Operator usage with better deployment, multiple ways to auto-scale, etc.

One gotcha on using AWS spot instances. Since every taskmanager interruption will cause a job restart it can be an issue using spot instances if you're running jobs with long initialization times, i.e. state heavy jobs. We use spot instances since our jobs are quick to restart in most cases - but it's been a pain when AWS reclaimed an instance in the middle of doing heavy backfills with multi-year historical data.

Thanks for the heads-up! By the way, I might have found a solution to S3 throttling: https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/deployment/filesystems/s3/#entropy-injection-for-s3-file-systems