Is there any "blessed" solution for doing zero-dow...
# troubleshooting
j
Is there any "blessed" solution for doing zero-downtime Flink deployments? What I've tried to do so far is something like this: 1. Make a 2nd Kafka topic to output the new data to 2. Make a 2nd deployment of the Flink job with the new changes and wait for it to catch up on all the data from the input topics 3. Once it's caught up ^ switch over the consumers to the new Kafka topic 4. Delete the old Kafka topic and Flink deployment. At the moment, each step is manual, which is not great. I could of course automate this process but it seems a bit tricky. You'd need: • A) An abstraction over Kafka topics - some central config that producers & consumers would have to look at to get their Kafka topic assigned (
my_topic.v1
for the old job,
my_topic.v2
for the new job) • B) An abstraction over Flink deployments so you could automatically have multiple deployments (
my_flink_deployment_v1
,
my_flink_deployment_v2
) which originate from different versions of the same code (We're using the k8s operator) So, questions: • Is there a better way to do this? ^ (maybe the flow is different/easier when using AWS managed Flink?) • If this is the way to do it, are there any existing libs that could help with automating this flow?
p
Hello there, Jan! There is FLIP-503 to bring blue/green deployments to k8s operator. But it's still not there.
j
Looks promising, thanks for letting me know!