Hi everyone, I am using Apache Flink 1.20.4, and I have the following high-availability requirement: I want to configure my Flink cluster so that if a job is running on one node (for example, Node 1) and that node suddenly fails, another node in the cluster automatically takes over the job from the latest available state. The new node should have access to all the required checkpoint/savepoint data so that the job can resume processing without data loss or manual intervention. What is the recommended HA architecture and configuration to achieve this? Specifically: • Which HA service should I use? • Where should checkpoints/savepoints be stored? • Which state backend is recommended? I do not want to use Kubernetes or HDFS. I have seen recommendations to use GlusterFS as shared storage for checkpoints and savepoints. Is GlusterFS a recommended approach for this setup, or is there a better alternative? Any suggestions or best practices would be appreciated.

While you haven't hinted that AWS is an option, would S3 be an option? We store our checkpoints/savepoints on S3 and with 1TB state restore it's "reasonable" fast (16 EC2 r6id.xlarge ~ 25 min) ...more EC2 the faster it is! e.g: 32 EC2 10 mins. Thus our setup: Zookeeper (ships with Flink) + S3 • 3 Node zookeeper Each cluster (scheduler-mode: REACTIVE + Job manager scheduler: Adaptive): • 2 Node job-manager (HA) • N nodes task-managers (EC2 memory + storage optimized r6id) • All nodes are individual EC2 instances. HDFS is great and first considered it (previously used with Hadoop), but discovered in my setup using S3, that S3 is not the limiting factor! We cap out the CPU and IOPS of our NVMe's during state restores. Thus, if we need fasters we either need (1) more disks or (2) faster disks. Never used GlusterFS but seems very similar to HDFS.

What other shared storage options are available for Apache Flink other than Kubernetes, HDFS, and S3 basically cloud?

I'm late here, but: GlusterFS will work, or any other shared filesystem that you can mount locally. You just have to make sure that the path to this mount is the same on all hosts that are running Flink.