Hi all — I’d like to initiate a discussion around a potential enhancement in Flink’s runtime resource management.
In long-running production clusters, we sometimes observe TaskManagers that repeatedly fail, flap on heartbeats, or exhibit degraded performance. Currently, Flink does not provide a cluster-level, scheduling-aware mechanism to temporarily quarantine such nodes. FLIP-224 introduced a blocklist mechanism, but it is primarily designed for speculative execution in batch workloads and does not serve as a general node governance framework.
I’m exploring the idea of introducing a lightweight Node Health Management layer at the ResourceManager level, with the following characteristics:
• Disabled by default and fully backward compatible
• Slot filtering before allocation (no scheduler redesign)
• Manual quarantine via REST API in phase 1
• Optional expiration and basic observability
The goal is not to replace Kubernetes/YARN controls, but to provide a runtime-level mechanism that is aware of Flink-specific failure semantics.
I’d really appreciate feedback on whether this direction makes sense within Flink’s architecture, and whether it is worth drafting a formal FLIP.
Thanks in advance for your thoughts.
https://lists.apache.org/thread/k22bgrqnvb552m64p54g2p1l2cjso2qc
https://docs.google.com/document/d/1-kRk01swcBaw6saBNJqoCVxD4vWfXiez-FaW_Oknz5c/edit?tab=t.0#heading=h.workco65f783