Hi Guys, After flink job submitted in EMRand if one of the nodes lost we are seeing below error even though other nodes are available to pick it up and also retry strategy is not working as well. Can some one help me here

org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException: Could not send message [RemoteFencedMessage(00000000000000000000000000000000, RemoteRpcInvocation(null.updateTaskExecutionState(TaskExecutionState)))] from sender [Actor[<akka.tcp://flink@ip-10-121-176-58.vpc.internal:42393/temp/jobmanager_2$Rs]]> to recipient [Actor[<akka://flink/user/rpc/jobmanager_2#-1101702724]]>, because the recipient is unreachable. This can either mean that the recipient has been terminated or that the remote RpcService is currently not reachable.

This may be related to a system constraint. What EC2 instance size and data size are you interacting with? Is this a batch or streaming job? Message and record sizes/number of tables? These kinds of errors, in my experience can happen when an EC2 instance/compute resource has capped out on system resources and is effectively ddos'd by the demand. For example, if you are at 100% memory utilization and disk usage on a worker node it may be impossible for that node to even create the network threads necessary to communicate. Usually the root cause for this kind of overload on a single node is data skew.

Hi @Jalil Alchy this is a streaming job and it’s happening when EC2 scaling happening(resizing), looks like yarn is not assigning job to available resources when one of the node down as part of scaling, we are trying to use restart strategy but seems like it’s not helping, is there a way we can make sure job assigned by available resources when scale down happens