Hello, I reduced one of my node of the yarn cluste...
# troubleshooting
n
Hello, I reduced one of my node of the yarn cluster and one of job started getting this error "Reaching max start worker failure rate" . It repeated itself every 6 mins and continued.
Copy code
akka.event.slf4j.Slf4jLogger$$anonfun$receive$1 akka.remote.ReliableDeliverySupervisor10:17:12.863 [flink-akka.actor.default-dispatcher-32] WARN  o.a.f.r.r.a.ActiveResourceManager - Reaching max start worker failure rate: 10 events detected in the recent interval, reaching the threshold 10.000000.
 org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager10:17:18.868 [flink-akka.actor.default-dispatcher-32] WARN  o.a.f.r.r.a.ActiveResourceManager - Reaching max start worker failure rate: 11 events detected in the recent interval, reaching the threshold 10.000000.
 org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager10:17:18.869 [flink-akka.actor.default-dispatcher-32] WARN  o.a.f.r.r.a.ActiveResourceManager - Reaching max start worker failure rate: 12 events detected in the recent interval, reaching the threshold 10.000000.
 org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager10:17:24.873 [flink-akka.actor.default-dispatcher-32] WARN  o.a.f.r.r.a.ActiveResourceManager - Reaching max start worker failure rate: 13 events detected in the recent interval, reaching the threshold 10.000000.
 org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager10:17:24.873 [flink-akka.actor.default-dispatcher-32] WARN  o.a.f.r.r.a.ActiveResourceManager - Reaching max start worker failure rate: 14 events detected in the recent interval, reaching the threshold 10.000000.
 org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager10:17:30.877 [flink-akka.actor.default-dispatcher-32] WARN  o.a.f.r.r.a.ActiveResourceManager - Reaching max start worker failure rate: 15 events detected in the recent interval, reaching the threshold 10.000000.
 org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager10:17:30.877 [flink-akka.actor.default-dispatcher-32] WARN  o.a.f.r.r.a.ActiveResourceManager - Reaching max start worker failure rate: 16 events detected in the recent interval, reaching the threshold 10.000000.
 org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager10:17:36.881 [flink-akka.actor.default-dispatcher-32] WARN  o.a.f.r.r.a.ActiveResourceManager - Reaching max start worker failure rate: 17 events detected in the recent interval, reaching the threshold 10.000000.
 org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager10:17:36
It continued till 110900 to finally fail. It means it took ~1 hr( 52 mins) I want it to fail fast. We lost the data for 52 mins. Is there any config to tweak this behaviour?