Hi, I'm unclear on the behaviour of session window...
# troubleshooting
t
Hi, I'm unclear on the behaviour of session windows. I'd like to use the example below where wallclock time is the real time of the events, and
timestamp
is the timestamp on the message by which the
SerializableTimestampAssigner
fetches the timestamp from the message payload. Assuming a 10s session window. Assume a reducer
Latest
which only send the latest message received in the window.
Copy code
keyedStream
     .window(EventTimeSessionWindows.withGap(TEN_SECONDS))
     .reduce(new Latest<>())
     .uid("LatestThrottle");
Copy code
Assuming the same Key on a 10s session window.
Wall Clock Time[T0].        M1[timestamp=0]
Wall Clock Time[T11].      M2[timestamp=11]
Wall Clock Time[T12].      M3[timestamp=10]
Wall Clock Time[T13].      M4[timestamp=11]
M1 opens the message window. No message flows downstream yet Does M2 closes the message window? Or does flink close the window using some sort of internal mechanism (timer?) so that M1 is released downstream at wall clock time T10? Does M2 flow because technically it was the message that caused the window to close and therefore the 'last' for the purposes of the reducer, or is it M1 that was within the 10s window and so M1 flows, but what happens to M2? See how M3 came in out of order, does this cause a new window to open, or was a new window opened by M2. I have too many questions because of the combinatorial situation above 😄 , so it's probably easier if somebody just states what the downstream output of that sequence is and why? 🙏
p
wallclock time is the real time of the events
Does real here means when it happened?
timestamp
is the timestamp on the message by which the
SerializableTimestampAssigner
fetches the timestamp from the message payload
So your watermark strategy uses timestamp, right?
Does M2 closes the message window? Or does flink close the window using some sort of internal mechanism (timer?) so that M1 is released downstream at wall clock time T10?
It will depend on your late events/out of orderness config.
See how M3 came in out of order, does this cause a new window to open, or was a new window opened by M2.
If M2 opens a new window, there a chance M3 would be on the same window of M2. I wrote this post here which may help The case for a Custom Window in Flink: Expanding your streaming use-cases I'm open to grab a coffee if you want to chat more about windows :D
t
Does real here means when it happened?
yep, as if when it happened you looked at your watch and noted the time
let me take a look at the post, thanks for the link 👍
🙌 1