Hi team qq.
I have a stream that we want to reduce over a sliding window by event time. The problem is we want every single time window to be accounted for, even if there's no events.
Currently the plan is to use a keyed process function. In the process function we
• Maintain a list of "windows". Basically a list of timestamped lists (where each list belongs to a time window)
• Any event that comes in gets added to all the windows they belong to
• If the event has a new key, then our context has another timer linked to this key
• Realistically we'd want every single key to always be there. However we may want to delete keys
◦ Either add in some manual intervention through perhaps a broadcast
◦ Or we just have a TTL for each key state (likely what I'd do)
• Can also allow late arrivals under this model
I believe this will work for our usecase, it shouldn't be overly daunting to set up. But
is there a more ergonomic way to do this? I've looked at how I can override some of the existing Window functionality to pull this off but I don't see a way. It feels like it should be very simple. I want a sliding window where the window creation is decoupled from element input. But there doesn't seem to be a cleaner way to implement this than above^. It'd be nice to use existing window features