Hi All! I am getting the following error in my log...
# troubleshooting
d
Hi All! I am getting the following error in my logs when my job starts from a checkpoint in Kubernetes. I was also able to find the oversized payload error messages as well showing the size of the incoming message and the max size (akka.framesize). I’m wondering if there is any way to debug the contents of the
TaskDeploymentDescriptor
? The only change we made was to switch to a new custom file sink that uses the Two-Phase-Commit. But the sink is running at-least-once and holds zero state, so I’m not sure why we’d get this issue.
Caused by: java.util.concurrent.TimeoutException: Invocation of [RemoteRpcInvocation(TaskExecutorGateway.submitTask(TaskDeploymentDescriptor, JobMasterId, Time))] at recipient [akka.tcp://flink@172.27.241.201:6122/user/rpc/taskmanager_0] timed out. This is usually caused by: 1) Akka failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase akka.ask.timeout.