Hi all — I’d like to initiate a discussion around ...
# dev
z
Hi all — I’d like to initiate a discussion around a potential enhancement in Flink’s runtime resource management. In long-running production clusters, we sometimes observe TaskManagers that repeatedly fail, flap on heartbeats, or exhibit degraded performance. Currently, Flink does not provide a cluster-level, scheduling-aware mechanism to temporarily quarantine such nodes. FLIP-224 introduced a blocklist mechanism, but it is primarily designed for speculative execution in batch workloads and does not serve as a general node governance framework. I’m exploring the idea of introducing a lightweight Node Health Management layer at the ResourceManager level, with the following characteristics: • Disabled by default and fully backward compatible • Slot filtering before allocation (no scheduler redesign) • Manual quarantine via REST API in phase 1 • Optional expiration and basic observability The goal is not to replace Kubernetes/YARN controls, but to provide a runtime-level mechanism that is aware of Flink-specific failure semantics. I’d really appreciate feedback on whether this direction makes sense within Flink’s architecture, and whether it is worth drafting a formal FLIP. Thanks in advance for your thoughts. https://lists.apache.org/thread/k22bgrqnvb552m64p54g2p1l2cjso2qc https://docs.google.com/document/d/1-kRk01swcBaw6saBNJqoCVxD4vWfXiez-FaW_Oknz5c/edit?tab=t.0#heading=h.workco65f783
👍 2