Antonio Davide Cali
02/25/2025, 3:44 PMEventTimeSessionWindows with a fixed gap.
I need, on other hand, to trigger a FIRE_AND_PURGE for closing a window after a fixed amount of time since the first event received for the window, no matter what.
So for doing so I'm implementing a Trigger
On a easy way, I'm trying on having exactly something like this.
where if gap in between the current event timestamp and the first one saved via the descriptor is greater than my gap, I just fire and purge the window.
@Override
public TriggerResult onElement(BigquerySnowplow element, long timestamp, W window, TriggerContext ctx) throws Exception {
// Get the timestamp of the first event in the window
ValueState<Long> firstEventTimestamp = ctx.getPartitionedState(descriptor);
if (firstEventTimestamp.value() == null) {
firstEventTimestamp.update(timestamp);
ctx.registerEventTimeTimer(timestamp + gapDuration);
} else if (timestamp - firstEventTimestamp.value() > gapDuration) {
return TriggerResult.FIRE_AND_PURGE;
}
// Continue adding elements to the window
return TriggerResult.CONTINUE;
}
This """kind of""" work.
{"startTime":"2025-02-20 15:12:06.065","endTime":"2025-02-20 15:12:25.605","firstEventTime":"2025-02-20 15:12:06.065","lastEventTime":"2025-02-20 15:12:18.605","sessionEventsCount":10}
{"startTime":"2025-02-20 15:12:06.065","endTime":"2025-02-20 15:12:25.605","firstEventTime":"2025-02-20 15:12:17.099","lastEventTime":"2025-02-20 15:12:17.099","sessionEventsCount":1}
{"startTime":"2025-02-20 15:12:25.992","endTime":"2025-02-20 15:12:34.346","firstEventTime":"2025-02-20 15:12:25.992","lastEventTime":"2025-02-20 15:12:27.346","sessionEventsCount":7}
My concern is the second window created where it is "correctly" created, but what I would actually expected is that.
1. startTime and endTime that accordingly just describe the window time, should be different from the first window
2. the second and the third window should be "merged" since the second one should have started a new window that events of the third one would have fallen into
I'm trying to figure out correctly how to properly understand what I'm doing wrong.
Any help would be very great gratitude thank youPedro Mázala
02/26/2025, 10:11 AMTriggerResult is probably controlling something outside of this scope. So how is this TriggerResult handled later on?
I'm wondering if having a keyBy doing this event wouldnt help. Because having different keys (you could add your window tuble to your key) would make sure Flink is in control of segmenting the stored data and you could rely on TTL.
Perhaps I'm missing something but happy to iterate over it 😄Antonio Davide Cali
02/26/2025, 10:17 AMEventTimeSessionWindows.withGap
KeyedStreamBigquerySnowplow, String eventsGroupedByUserID = eventsSource.keyBy(BigquerySnowplow::getClientId);
DataStreamBigQueryEventsSession sessions = eventsGroupedByUserID
.window(EventTimeSessionWindows.withGap(Duration.ofSeconds(7)))
Now this works as intended, but I need to add a custom trigger on top, because I don't want this window to grows indefinitely.
I want this window to close after a fix amount of time no matter what since first event for this user arrive
Something like TimeGapTrigger.of(Duration.ofSeconds(10))
DataStreamBigQueryEventsSession sessions = eventsGroupedByUserID
.window(EventTimeSessionWindows.withGap(Duration.ofSeconds(7)))
.trigger(TimeGapTrigger.of(Duration.ofSeconds(10)))
This Trigger is a Custom Trigger that I'm trying to implement, that saves the very first event timestamp encountered for this user, and then check if gapDuration passed, and in case close the window.Pedro Mázala
02/26/2025, 11:21 AM@Override
public TriggerResult onElement(BigquerySnowplow element, long timestamp, W window, TriggerContext ctx) throws Exception {
ValueState<Long> firstEventTimestamp = ctx.getPartitionedState(descriptor);
if (firstEventTimestamp.value() == null) {
firstEventTimestamp.update(timestamp);
ctx.registerEventTimeTimer(timestamp + maxSessionLengthMillis);
}
/*
* We do NOT fire the window right away here because
* we only want to fire after the maximum session length passes,
* or if the inactivity gap triggers a session window close
* at the operator level.
*/
return TriggerResult.CONTINUE;
}
/**
* Called when an event-time timer you've registered fires.
*/
@Override
public TriggerResult onEventTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {
// If an event-time timer fires, it implies that the (firstEventTs + maxSessionLength) has been reached.
// We want to fire AND purge the window immediately.
return TriggerResult.FIRE_AND_PURGE;
}
/**
* If you don't need any special handling on processing-time timers,
* you can simply ignore or do nothing here.
*/
@Override
public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) {
return TriggerResult.CONTINUE;
}
I think the issue was on closing the session on the new event insted on the timerAntonio Davide Cali
02/26/2025, 11:35 AMonEventTime method overwritten as you mentioned for closing the window.
Just to clarification I'm running the job with the following Parameter
Window: EventTimeSessionWindows.withGap(Duration.ofSeconds(7)
CustomTrigger: TimeGapTrigger.of(Duration.ofSeconds(10))
Removing the FIRE_AND_PURGE on the onElement methods gives me now the following output:
WINDOW1: {
"startTime": "2025-02-20 15:12:06.065",
"endTime": "2025-02-20 15:12:25.605",
"firstEventTime": "2025-02-20 15:12:06.065",
"lastEventTime": "2025-02-20 15:12:18.605",
"sessionEventsCount": 11
}
WINDOW3: {
"startTime": "2025-02-20 15:12:25.992",
"endTime": "2025-02-20 15:12:34.346",
"firstEventTime": "2025-02-20 15:12:25.992",
"lastEventTime": "2025-02-20 15:12:27.346",
"sessionEventsCount": 7
}
What we can notice is that that compared to the previous results we don't have anymore the WINDOW2
{
"startTime": "2025-02-20 15:12:06.065",
"endTime": "2025-02-20 15:12:25.605",
"firstEventTime": "2025-02-20 15:12:17.099",
"lastEventTime": "2025-02-20 15:12:17.099",
"sessionEventsCount": 1
}
Because this event felt in the WINDOW1.
But taking a look at it, we can see
{
"startTime": "2025-02-20 15:12:06.065",
"endTime": "2025-02-20 15:12:25.605",
"firstEventTime": "2025-02-20 15:12:06.065",
"lastEventTime": "2025-02-20 15:12:18.605",
"sessionEventsCount": 11
}
What I need to have is that
startTime should be 2025-02-20 15:12:06.065 (as it is)
but endTime should be 2025-02-20 15:12:16.605 (compare to 151225),
This is because what I'm trying to achieve with my custom trigger is that
a window should be closed after 10 seconds (TimeGapTrigger.of(Duration.ofSeconds(10)) ) from startTime.
But it doesn't.
It actually close only when there are no more message after 7 seconds (EventTimeSessionWindows.withGap(Duration.ofSeconds(7) ) since last message received.Antonio Davide Cali
02/26/2025, 11:39 AM{
"startTime": "2025-02-20 15:12:06.065",
"endTime": "2025-02-20 15:12:25.605",
"firstEventTime": "2025-02-20 15:12:17.099",
"lastEventTime": "2025-02-20 15:12:17.099",
"sessionEventsCount": 1
}
contains exactly the only records at 151217 that I have for this user, after the 10 fixed seconds of closing windows I put in my trigger.
Because the window start at 151206, 10 seconds passed, it closed the windows because that's what I want, and it should create a new one (like it does).
But once again the startTime and endTime for this new window created is wrong.
Plus the Window3 should be merged with Window2 because Window2 is now the "leader" of this specific timespan (151217-151227)Pedro Mázala
02/26/2025, 12:36 PMAntonio Davide Cali
02/26/2025, 1:08 PM