I could use some help conceptualizing the right way to frame a problem in Flink. I can’t use the complex event processing system due to the nature of other requirements. Anyway, say I have a data stream of events where each event has an event time and a Boolean field (call it A for simplicity). I’d like to send an alert out if we ever see a continuous stretch of events where A is true for at least five seconds. Put another way, alert if we did not see an event where A was false within five seconds of an event where A was true. My gut tells me that some combination of a sliding window and custom trigger is what this calls for, but I can’t quite nail it down.

Since its more than one dimension, check this Flink-CEP (Complex Event Processing) - https://nightlies.apache.org/flink/flink-docs-master/docs/libs/cep/

I don’t believe I can use complex event processing. I’m making a system for users to configure dynamic rules to monitor data from our systems.

not sure of pyflink

Rules enter the system as part of a control stream that is broadcasted and then connected to the input stream of data to process. The motivation for this pattern is similar to what was developed in https://github.com/brggs/dynamic-flink

saw this video few months back, hope it help -

Thanks, I’ll check it out! I think the problem is much easier if we only cared about the latest data with respect to event time and then gets more challenging if we want to alert on older data coming in out of order.

smaller windows should be ok, like temp > 100C for 10 min , if you are watching for 1-2 hrs, its tricky

Do you think session windowing would make more sense to more dynamically handle the variable nature of the throughput?

Yeah, Sliding is best bet, depends on time interval

Sliding windows aren’t ideal for this — because their behavior doesn’t precisely match your requirements. The issue is that sliding windows are aligned to the epoch, whereas in your example, you’d like to consider an interval of 5 seconds relative to the first event where A is true. You could implement custom windowing that handles this, but this will be rather more straightforward to implement using a process function.

Yes, that was my concern. Getting the windows right felt like it could miss things.

If you try to use windows, you’ll basically end up in the same place. I think you’f need to use GlobalWindow with a custom Trigger and Evictor — which is a mess to deal with. A KeyedBroadcastProcessFunction will probably lead to a cleaner implementation — and besides, you can’t broadcast into a window.