We are currently operating a Flink deployment on Kubernetes, with high availability (HA) configured using Kubernetes-based HA services. We're exploring approaches for geo-redundancy (GR) to ensure disaster recovery and fault tolerance across regions or clusters.
So, the questions could be:
Is there any established or recommended approach in Flink to support geo-redundancy — particularly for stateful jobs with RocksDB state backend and checkpointing to a remote object store (like S3)?
• In our scenario, we plan to maintain two geo-distributed Flink clusters, and in the event of a disaster, bring up the same job on the standby (DR) cluster using a retained checkpoint from a replicated S3 bucket.
• Are there any known best practices, pitfalls, or limitations with respect to using this model for disaster recovery?
We are currently evaluating the use of cross-region replicated S3-compatible storage to store checkpoints. Our intention is to:
• Enable automatic or manual failover by starting the same Flink job on the GR cluster from the replicated checkpoint.
• Use incremental RocksDB state, with exactly-once semantics ensured via Kafka source/sink integration.
A key concern here is:
• If the most recent checkpoint is not yet fully replicated to the second S3 cluster when a failover happens, how can we safely restore while maintaining exactly-once semantics?
Is there any recommendation or mechanism (either available in Flink or implemented externally) to:
• Delay marking a checkpoint as complete/committed until its files are successfully replicated, or
• Restore from the latest fully replicated and durable checkpoint only, with confidence that state and offsets are consistent?