Hi all, our Flink apps are currently deployed on Kubernetes (vanilla) without utilizing flink-k8-operator. We've been encountering issues where the JobManager gets stuck & becomes unresponsive due to OOM, requiring manual intervention for restarts. Our objective is to automate this process. Given our Kubernetes deployment, I've been exploring the possibility of leveraging any existing Flink 1.9 Rest APIs or building a wrapper service around Flink REST API that would be configured as liveness probe for JobManager pod. Has anyone encountered or resolved a similar issue before? If yes, what was the approach on handling such issues?

Any reason why you are not upgrading your Flink version?

It's there in the engineering backlogs but not prioritized for now. It would be the last option as I can see

/jobs/:jobid/clientHeartbeat

REST API available for latest versions. https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobs-jobid-clientheartbeat

Given the amount of work that has been gone in making Flink run better on Kubernetes, I would highly advise upgrading to a later Flink version

Are u suggesting on upgrading Flink & migrating to flink-k8s-operator?

Definitely an upgrade of Flink. It's not supported in the community, as it only supports the last 2 minor versions (1.16 and 1.17, pretty soon only 1.17 and 1.18)