Hii All I have a basic doubt around watermark gene...
# troubleshooting
r
Hii All I have a basic doubt around watermark generation. Let's say I have a kafka topic with 10 partitions and if I provide source parallelism to be 15, with idleTimeout of 30 mins. In this case, how are the watermarks generated ? 1. Does it hold back watermark due 5 (15-10) idle slots till 30 mins ? 2. Or it identifies that kafka partition count is 10 and hence progress the watermark. I am observing that it shows
No Watermark (Watermarks are only available if EventTime is used)
for quite some time and then starts generating watermark.
d
The KafkaSource will indeed wait for the idle timeout before progressing the watermark.
r
Yes David, it holds back the watermark till idleTimeout passes. So does it mean we must provide source parallelism equal to kafka partitions, and never more than that ?
If you are using
BoundedOutOfOrderness
, then you can specify like
Copy code
val watermarkStrategy: WatermarkStrategy[Option[T]] = WatermarkStrategy
      .forBoundedOutOfOrderness(Duration.ofSeconds(maxOutOfOrdernessInSecs))
      .withIdleness(Duration.ofSeconds(idlenessInSecs))
You can apply it on any WatermarkStrategy using
.withIdleness