Hello, is there a way to process watermarks in a `...
# troubleshooting
s
Hello, is there a way to process watermarks in a
BroadcastConnectedStream
? 🧵
I am trying to process watermarks in a
BroadcastConnectedStream
. However, I am not able to find any direct way to handle watermark events, similar to what we have in
processWatermark1
in a
KeyedCoProcessOperator
.
In the context of the example given in “A Practical Guide to Broadcast State in Apache Flink”, I have a user actions stream and a pattern stream. The pattern stream is broadcast and connected with the user actions stream. The result is a
BroadcastConnectedStream
. I want to handle user action events and pattern evens in this stream. In addition, I want to use a
processWatermark
function to perform an action in response to watermark events.
The problem is that a
BroadcastConnectedStream
has only
process()
function, no
transform()
, that takes a
(Keyed)BroadcastProcessFunction
. A
BroadcastProcessFunction
allows only to process elements, it doesn’t provide the interface to process watermarks. In contrast, a
ConnectedStream
(without broadcast) provides a transform function, which takes in an operator that provides a way to process watermarks.
d
If this is a KeyedBroadcastProcessFunction, then as a workaround you can, in the processElement and onTimer methods, register an event time timer for
currentWatermark + 1
. If you do that for every incoming event (and for every timer firing), you'll be guaranteed that the timer is called every time the watermark advances (for every key).
s
Thanks David, that worked for me.
d
Glad to hear it. Since timers are deduplicated, this should be a reasonable solution.
👍 1