HI Everyone, Today we got one error in production ...
# troubleshooting
n
HI Everyone, Today we got one error in production env, where I got see the below error log
Copy code
akka.remote.OversizedPayloadException: Discarding oversized payload sent to Actor[akka.tcp://flink@10.220.51.56:6122/user/rpc/taskmanager_0#1423510823]: max allowed size 10485760 bytes, actual size of encoded class org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation was 14835748 bytes.
After above issue I started getting this exception,
Copy code
ERROR akka.remote.EndpointWriter [] - Transient association error (association remains live)
Later, what I saw Flink job was in restarting mode for few hours, until i went and manually killed the job to do so, FYI I am using flink
1.19.0
Any help would be really helpful, so that same wont get repeated in future. Thanks in advance.
f
sorry if it is too obvious but for the error you show you have a problem with the size of your payload, the size of your payload is 14835748 bytes and the maximum size allowed is 10485760 bytes, I am not a flink expert :)
n
HI Thanks for the help, but when I tried increasing max payload size allowed, still it did not work. btw I got this ticket raised at flink dashboard https://issues.apache.org/jira/browse/FLINK-34430 says issue with rocksDB sst file size, instead of payload. If anyone can help on this
Copy code
pekko.ask.timeout: 600s
  pekko.client.timeout: 600s
  pekko.framesize: "16835800b"