Hi team, in our flink job we are taking checkpoint every 5 mnts. Is there any way to delay shutdown of a taskmanager (when taskmanager gets SIGTEM signal) until next checkpoint is taken or force checkpoint just before shutdown? The context is, in our kubernetes cluster we recycle nodes older than 7d. Candidates for recycling is discovered in a 20mnts interval so we don't know when exactly node recycling is going to happen. If task manager is killed just before next checkpoint we need to reprocess huge amount of events which we are trying to avoid. We know a node can go down unexpectedly but we are atleast trying to improve expected node recycling scenario. We are using flink kubernetes operator for job deployment. Thanks!

Wouldn’t it then make more sense to restart your jobs every 6 days? 😅

Unfortunately nodes are not same age always. So recycling is not happening at constant time. A node recycle can happen in 20 mnts if a particular node just became 7d old 😞

That’s tough. There is little that Flink can do in these cases

Thanks for responding!