Can someone please share the idea if they have a d...
# troubleshooting
s
Can someone please share the idea if they have a disaster recovery support for flink cluster in production and how the exactly-once is being managed in such setups?
p
Hello there! Are you looking into disaster recovery from which perspective? I worked on a disaster recovery plan before. The first step was defining the kind of disasters we were willing to have a plan for. This may help you to find out where your attention is more needed
s
This. Some dimensions to think along. • The scale of the failure. Single pod (in k8s)/single node/single rack/single AZ/single datacenter • What your sharded/keyed state in Flink looks like today. • What are your SLA requirements for uptime/latency?
👍 1
p
Those are awesome dimensions. you can think of: • what happens if I loose state? ◦ can I rebuild it? ◦ should I create a external state backup/snapshot? • What happens if there is a breaking change while upgrading? ◦ Do I use kryo? ◦ Does the operators have UID set? • How long Flink takes to go from DEPLOYING to RUNNING? • What happens when my sink is down? The tip here is to think on the most possible cases first and grow from there.
s
We are currently operating a Flink deployment on Kubernetes, with high availability (HA) configured using Kubernetes-based HA services. We're exploring approaches for geo-redundancy (GR) to ensure disaster recovery and fault tolerance across regions or clusters. So, the questions could be: Is there any established or recommended approach in Flink to support geo-redundancy — particularly for stateful jobs with RocksDB state backend and checkpointing to a remote object store (like S3)? • In our scenario, we plan to maintain two geo-distributed Flink clusters, and in the event of a disaster, bring up the same job on the standby (DR) cluster using a retained checkpoint from a replicated S3 bucket. • Are there any known best practices, pitfalls, or limitations with respect to using this model for disaster recovery? We are currently evaluating the use of cross-region replicated S3-compatible storage to store checkpoints. Our intention is to: • Enable automatic or manual failover by starting the same Flink job on the GR cluster from the replicated checkpoint. • Use incremental RocksDB state, with exactly-once semantics ensured via Kafka source/sink integration. A key concern here is: • If the most recent checkpoint is not yet fully replicated to the second S3 cluster when a failover happens, how can we safely restore while maintaining exactly-once semantics? Is there any recommendation or mechanism (either available in Flink or implemented externally) to: • Delay marking a checkpoint as complete/committed until its files are successfully replicated, or • Restore from the latest fully replicated and durable checkpoint only, with confidence that state and offsets are consistent?
Hello all, any help here would be appreciated. Thanks
k
Also curious how others are doing it as we are running into the same situation! 👀 Anything from community will be very appreciated! 🙏
p
The work I've done with disaster recover is IP from my prev company, so not shareable. Geo-redundancy may be achieved with k8s instead of Flink IMO. If you're worried about that you may have distributed nodes among different zones. On the s3 side you may think about bucket replication in a sense. Although s3 has been quite stable in the last few years. > A key concern here is: > • If the most recent checkpoint is not yet fully replicated to the second S3 cluster when a failover happens, how can we safely restore while maintaining exactly-once semantics? Depending on the state size this may be an issue. You also need to consider how often you're running checkpoints to understand the likeability for this to happen. > Is there any recommendation or mechanism (either available in Flink or implemented externally) to: I'd say you most probably will find this outside of flink