I want to build a signal smoother in Flink - Like ...
# troubleshooting
t
> I want to build a signal smoother in Flink - Like a latch with 20 second expiry when no 'OPEN' happens. > • The input is a stream of events, each with a field
state
which is either { OPEN, CLOSED } and a timestamp for EventTime processing. > • Events arrive continuously and may flap between open and close very rapidly. > > When an Open event is received: > • Emit open immediately (don't wait) - this ensures potential problems trigger alerts right away. > When a close event is received. > • Do not emit it immediately. > • Instead start a 20 second timer > • If no new open arrives during those 20 seconds, emit the close when the timer fires. > If another open arrives before the timer fires > • Cancel the timer > • The close is suppressed.... > • The open is dropped but this is ok because the state is already open. Is there a pattern for this? EventSessionWindows feel not quite right as they do not allow immediate emitting of Opens.
Picture tells a thousand words, through perhaps not to scale. See how the various bumps are resolved to a “safe” OPEN state. And closed state isn’t present till 20 seconds after the last blip signal
l
The logic you defined seems like it would work well with using the KeyedProcessFunction. You can create a POJO with the current state (perhaps a boolean with True for Open and False for Closed) and a last updated timestamp using the event's timestamp. > When an Open event is received: > • Emit open immediately (don't wait) - this ensures potential problems trigger alerts right away. You create a ValueState instance and can populate the last updated timestamp using the event's timestamp, and set isOpen to true. Emit the event via the processElement() collector. > When a close event is received. > • Do not emit it immediately. > • Instead start a 20 second timer > • If no new open arrives during those 20 seconds, emit the close when the timer fires. If the current state has a non null value, and isOpen is False, then register and event time timer of the event's time + 20 seconds. In the onTimer() method you can emit the closed event and delete the state object. > If another open arrives before the timer fires > • Cancel the timer > • The close is suppressed.... > • The open is dropped but this is ok because the state is already open. If an open arrives, then: • update the state's isOpen value to true • delete the event time timer (
ctx*.*timerService*().*deleteEventTimeTimer*(*timestampOfTimerToStop*);*
) . You can get the timestamp to use. by adding 20 seconds to the state's last updated time value. • update the state's last updated time to the incoming open event's event time.