Hey, We recently noticed a Flink application (1.15.0) that was running successfully for a few months that started failing with:

Transient association error (association remains live)
akka.remote.OversizedPayloadException: Discarding oversized payload sent to Actor[<akka.tcp://flink@10.137.128.14:50100/user/rpc/taskmanager_0#-1857979235>]: max allowed size 125829120 bytes, actual size of encoded class org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation was 218273805 bytes.

We tried increasing the “frame.size” Flink configuration but every time it failed with a larger size of RemoteRpcInvocation. The failure is in the SubmitTask call from the JobManager to the TaskManager. And we also tried to revert to a version that used to be working okay before, which has now failed because of the same error. Could you please provide some guidance on how to solve this issue? All our efforts searching for an answer have not been successful in highlighting the potential causes.

hi emily! have you had any luck with this? a few of our 1.15+ jobs have recently started failing with this same error. other jobs have been running for a year or so without it…

Sorry @Trystan I only just saw your reply. So what happened with our application was the application somehow got into a weird state, and the state file just keeps increasing everytime when we restart. We decided to restart the app with an empty state, and the issue was fixed afterward(the size of the state file stopped increasing).

that’s actually similar to what we’re seeing periodically as well