Hello team! We are using

watermark latency

in our pipeline to track e2e latency (Is it the correct metric to use when using

metrics.latency.granularity: "single"

? 🤔). Sometimes, latency is negative, what would it mean? In flink code watermark latency is defined as

processing_time - watermark

. Another question we have is that when using python udfs, PythonGroupAggregate Operator is used, which doesn’t track

watermark latency

. How we can use this metric with python? P.S. we are using flinnk 1.17

The watermarkLatency metric is only defined on a few of the built-in window operators. If you want a similar metric to be available elsewhere, you'll have to define it yourself.

metrics.latency.granularity

is unrelated to the watermarkLatency metric -- that relates to the more general purpose latency metrics Flink provides (they can measure latency between the sources and all other operators). It doesn't seem possible that the latency would be negative, unless you have a buggy custom watermark strategy.