How does

jobmanager.execution.failover-strategy=region

work? If I have two task pipeline A->B and I run it with parallelism of 5. Suppose 2 of the tasks fail (type B), will flink restart only those 2 tasks(of type B) or 4 tasks (both A & B) or all 10 tasks?

First of all, it should be clear that if a task fails, all tasks in the

pipelinedRegion

where it is located will be restarted. The formation of

pipelinedRegion

is related to the connection mode of edges 1. If A ->B is

ALL_TO_All

edges, all 10 tasks are in a single pipelined region. 2. If A>B is

PointWise

edge, the topology you described will form five pipelined regions. Each region will contain a parallel instance of A ->B.

Interesting, I didn't know about pointwise and all-to-all. I am using rebalance() operation between A and B so I am assuming my pipeline is connected using all_to_all. Would be nice if there was a way for flink to mix it up and send it to a constant number of B tasks which is calculated automatically based on load. That way not all tasks would need a restart. The only reason I do rebalance is because my parallelism is higher than required kafka consumers so rebalance helps redistribute the messages evenly. Thank you for answering 🙂