Quick question that’s tangentially related to Flink that I thought I’d pose to the audience here as there’s likely someone in the audience that has dealt with it previously. I’m currently trying to configure a Prometheus-based alert within Grafana to monitor Kafka consumer lag (

kafka_consumergroup_lag

) for one of my Flink jobs to determine when something upstream may have failed. This particular job doesn’t commit offsets at periodic intervals and the point of degradation isn’t a static value that might apply across all environments (so I can’t simply use something like

kafka_consumergroup_lag{topic="topic-for-flink"} > $someThreshold

). Does anyone have any ideas on this front? I’m basically just trying to set up an alert if my consumer lag for this particular job purely increases for an hour. I’ve tried a few different approaches involving derivatives, etc. but haven’t had much luck thus far.