Does anyone know how i might catch a org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException? The motivation is that I want to integrate that failure with my organizations alarming system.

Could try frequent polling of the exception from the REST endpoint? https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobs-jobid-exceptions

Hmmm if possible I’d like to do a “push” of a notification when this happens as this is a rather rare type of failure

Ah I see, the failure is triggered from the scheduler itself, I am not aware if we could hook any sort of callback from there tbh.

got it, thanks!

https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#availability We used the status metric for jobs. Whenever the status of a job is stuck in initializing state for a period of time, we sent out an alert

nice! this might be exactly what we need

@Varun Sayal @Sameer Chandra Could you please share on how did you solve this. We have similar situation where JobManager gets stuck due to OOM & always require manual intervention on restarts. We want to automate this & since we have Kubernetes architecture, I've been researching if we could use any existing Flink 1.9 Rest APIs or build a wrapper service on any existing REST API for our JM's K8 liveness probe.

Hey Deepanshu, the way we went about it was to monitor the JobManager metric for the number of available taskmanagers. We then do an aggregation to see if that number gets down to 0, if it does we alarm. We pump out those metrics using a jmx connector to our organizations alarming stack,