Hi, I have a flink deployment running in Kubernetes, where I’m seeing this error in the job manager for the different task:

TaskManager (example-taskmanager-1-1 @ 10.1.13.217 (dataPort=44647)) not responding after a rpcTimeout of 10000 ms

Also the taskmanager pods are contantly restarting after receiving SIGTERM.

2023-07-18 13:12:26,748 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2023-07-18 13:12:26,750 INFO org.apache.flink.runtime.state.TaskExecutorStateChangelogStoragesManager [] - Shutting down TaskExecutorStateChangelogStoragesManager.
2023-07-18 13:12:26,750 INFO org.apache.flink.runtime.filecache.FileCache         [] - removed file cache directory /tmp/flink-dist-cache-1a96228d-6862-419e-b9ec-bbefc102e5e1
2023-07-18 13:12:26,752 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl   [] - FileChannelManager removed spill file directory /tmp/flink-netty-shuffle-a069bf85-cde4-4c7b-a95f-c6ac6c85adef
2023-07-18 13:12:26,752 INFO org.apache.flink.runtime.io.disk.FileChannelManagerImpl   [] - FileChannelManager removed spill file directory /tmp/flink-io-8f51c6bf-cfe0-47c6-92ba-186bd0773393
2023-07-18 13:12:26,753 INFO org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager [] - Shutting down TaskExecutorLocalStateStoresManager.
2023-07-18 13:12:26,754 INFO org.apache.flink.runtime.blob.PermanentBlobCache       [] - Shutting down BLOB cache

The only information I found about this exception, that it could be related to OOM issues. I’ve already increased the resources of the taskmanager and I don’t experience the same behaviour if running with a smaller dataset. Is it possible that I still have resource issues or could this

rpcTimeout

indicate any other issue? Appreciate any inputs.

looking into this I would say that something else on the server stops your application. I believe that it would be another reason if you process takes too much Memory from the OS. Nevertheless, if you suspect it could be due to the memory issues, I would double check the TM configuration options: here

Have you checked the details of the pods that are restarting (

kubectl describe pod …

)? Typically I’d see OOM as a reason for termination in this case.

Restarting was not a good word to use, the jobmanager just spawns a new taskmanager (example-taskmanager-1-1, example-taskmanager-1-2, etc.) I can't see anything OOM related with the

describe pod

command, only that the error code is 239