Hi all,
We are running flink batch jobs which read data from kafka and upload them to s3 (basically they ingest data from kafka). We want to configure time based kafka lag alerts via prometheus i.e. want to get alerted if the lag is more than say 10 mins. Upon exploring metrics emitted by flink kafka source, we found 'currentEmitEventTimeLag' to be apt for our use case. However, since the jobs are not continuous in nature, the metric emitted in case of jobs which finish quickly (in seconds) doesn't get captured. Even if we tweak scrape interval in prometheus, there are chances that the metric is not captured unless we set the scrape interval to 1s which might overwhelm the prometheus server.
Are there any ways to handle this specifically for batch jobs? Any other approaches/suggestions are also welcome.
Thanks in advance!