Hi Team, We are seeing a weird issue with our stat...
# troubleshooting
s
Hi Team, We are seeing a weird issue with our stateful Flink Job. We see that the flink container is exceeding the allocated
process.size
/container limit by 2GB and is getting killed by the K8S, as it exceeds the defined K8s container limits. We do not see any issues with Heap or Network Memory exhaustion as such, we have plenty of buffer memory still left. What could be the area which is causing this extra memory request to k8s? How we can debug and find out this area? Also, we see this pattern repeat every 4-5 hours once but in seprate TMs.
p
You probably have some memory leak issue. What is the graph of the momory on those TMs?
s
We have checked the Heap usage for any application-related memory leak, however, we see that we have ample heap free when we encounter the crash. Also, the crash is initiated by the K8S memory manager, when it exceeds the container limit 100%. We are unable to understand, why this would happen?
p
We can see GC kicking every now and then but some stuff remain in the memory tho. It may be related to some leak. Either on your app or some code you’re using. This may help