Hi All, we are exploring technologies to support s...
# troubleshooting
s
Hi All, we are exploring technologies to support stream processing use cases. I am seeking best practices and guidance for leveraging Flink to support tier0 and tier1 availability of application within a multi-region active standby disaster recovery strategy. What options are available for running and restarting stateful Flink jobs in a failure region, given that Kafka topics are replicate in an active-active mode and that consumer groups and offsets differ across regions? Specifically, how can state be replicated to a failover region and stream processing Flink jobs be resumed with minimal or no downtime? (Kafka offsets are different)
d
Hi Sree, I would say as far as technology for stream processing use cases if you want the best performance and windowing capabilities its Flink hands down! There is simply no other stream processing systems that come close to its windowing capability and dual stream/batch mode unified api
Now as to your question regarding multi-region & failover you definitely will need a plan for this with any stream processing system. #1 You must ensure regular save points of Flink Jobs. These can be store in high availability data lakes S3, GCP Azure Blob etc if need. They need to be taken regularly. Thats key.
These can be accessed from any region.
#2 Use a HA State Backend like RocksDB with regular backups enabled & monitored!!!
Deploy Flink on Kubernetes or similar orchestration that is multi-region in nature.
## Implement a restart strategy. Default in indefinitely I think you can configure with fixed amount of delay as well.
One possibility ios
Copy code
FixedDelayRestartStrategy
Now while you mentioned Kafka as separate it’s kind of not separate!!!
Because you must carefully manage the Kafka offsets and consumer groups.
You will probably want consumer group Id per region for example.
You also need to manage offset synchronization across regions as needed.
And you must absolutely have a plan for offset reset in the event of a failover.
Finally Monitoring/Alerting is key to make it all work. In summary you are looking at a Kafka/Flink system where the streaming is done in Flink but is not totally independent of the offset and consumer group management. But using these strategies you can achieve your goals using the two most widely deployed solutions for messaging and streaming. Go with the best!!!
Oh and the monitoring is also to automate the failover process or at least its initialization as needed.
s
@D. Draco O'Brien Thank you for sharing your inputs. We aim to use Flink for tier-0 and tier-1 applications with minimal downtime. In the event of a regional outage, such as "us-east-1" being down, we plan to restart the jobs in "us-west-1." Since our Kafka cluster operates in active-active mode, the data will be available; however, the consumer groups and offsets will differ between regions. 1.While we can implement logic to map offsets, how do we update Flink savepoints accordingly? 2.Is rebuilding the state the only option, or are there alternatives? 3.How can we restart Flink applications with the correct state, ensuring that reprocessing duplicates does not corrupt the state?