Terabyte
05/14/2025, 9:00 PMtimestamp is the timestamp on the message by which the SerializableTimestampAssigner fetches the timestamp from the message payload. Assuming a 10s session window. Assume a reducer Latest which only send the latest message received in the window.
keyedStream
.window(EventTimeSessionWindows.withGap(TEN_SECONDS))
.reduce(new Latest<>())
.uid("LatestThrottle");
Assuming the same Key on a 10s session window.
Wall Clock Time[T0]. M1[timestamp=0]
Wall Clock Time[T11]. M2[timestamp=11]
Wall Clock Time[T12]. M3[timestamp=10]
Wall Clock Time[T13]. M4[timestamp=11]
M1 opens the message window. No message flows downstream yet
Does M2 closes the message window? Or does flink close the window using some sort of internal mechanism (timer?) so that M1 is released downstream at wall clock time T10?
Does M2 flow because technically it was the message that caused the window to close and therefore the 'last' for the purposes of the reducer, or is it M1 that was within the 10s window and so M1 flows, but what happens to M2?
See how M3 came in out of order, does this cause a new window to open, or was a new window opened by M2.
I have too many questions because of the combinatorial situation above 😄 , so it's probably easier if somebody just states what the downstream output of that sequence is and why? 🙏Pedro Mázala
05/15/2025, 8:59 AMwallclock time is the real time of the eventsDoes real here means when it happened?
So your watermark strategy uses timestamp, right?is the timestamp on the message by which thetimestampfetches the timestamp from the message payloadSerializableTimestampAssigner
Does M2 closes the message window? Or does flink close the window using some sort of internal mechanism (timer?) so that M1 is released downstream at wall clock time T10?It will depend on your late events/out of orderness config.
See how M3 came in out of order, does this cause a new window to open, or was a new window opened by M2.If M2 opens a new window, there a chance M3 would be on the same window of M2. I wrote this post here which may help The case for a Custom Window in Flink: Expanding your streaming use-cases I'm open to grab a coffee if you want to chat more about windows :D
Terabyte
05/15/2025, 9:00 AMDoes real here means when it happened?yep, as if when it happened you looked at your watch and noted the time
Terabyte
05/15/2025, 9:01 AM