hi all! I am trying to understand why the watermark in my Flink job is falling very far behind the current processing time. (On average seems to oscillate between 2 to 6 minutes behind) • I'm performing stream to stream join via the use of KeyedCoProcessFunction & firing event time based timers • Both stream sources are kafka topics with plenty of data in them. ◦ In test environment 36 & 12 partitions ◦ In prod environment 480 & 360 partitions • Watermarking Strategy for both topics is configured like so

forBoundedOutOfOrderness(Duration.ofSeconds(5))

• There is no backpressure, and the kafka consumer lag for both topics is consistently close to 0 • Checkpoints are stored in s3 and are taken every 60s, checkpoint duration is < 1s • Running Flink 1.14 I'm looking for some pointers on how to debug the problem & in general what could explain this behavior