Hello team, I hope it's not rude but I'm trying to...
# troubleshooting
a
Hello team, I hope it's not rude but I'm trying to figure out how to correctly move on this. Scenario, I am using a
EventTimeSessionWindows
with a fixed gap. I need, on other hand, to trigger a
FIRE_AND_PURGE
for closing a window after a fixed amount of time since the first event received for the window, no matter what. So for doing so I'm implementing a Trigger On a easy way, I'm trying on having exactly something like this. where if gap in between the current event timestamp and the first one saved via the descriptor is greater than my gap, I just fire and purge the window.
Copy code
@Override
    public TriggerResult onElement(BigquerySnowplow element, long timestamp, W window, TriggerContext ctx) throws Exception {
        // Get the timestamp of the first event in the window
        ValueState<Long> firstEventTimestamp = ctx.getPartitionedState(descriptor);

        if (firstEventTimestamp.value() == null) {
            firstEventTimestamp.update(timestamp);
            ctx.registerEventTimeTimer(timestamp + gapDuration);
        } else if (timestamp - firstEventTimestamp.value() > gapDuration) {

            return TriggerResult.FIRE_AND_PURGE;
        }

        // Continue adding elements to the window
        return TriggerResult.CONTINUE;
    }
This """kind of""" work.
Copy code
{"startTime":"2025-02-20 15:12:06.065","endTime":"2025-02-20 15:12:25.605","firstEventTime":"2025-02-20 15:12:06.065","lastEventTime":"2025-02-20 15:12:18.605","sessionEventsCount":10}

{"startTime":"2025-02-20 15:12:06.065","endTime":"2025-02-20 15:12:25.605","firstEventTime":"2025-02-20 15:12:17.099","lastEventTime":"2025-02-20 15:12:17.099","sessionEventsCount":1}

{"startTime":"2025-02-20 15:12:25.992","endTime":"2025-02-20 15:12:34.346","firstEventTime":"2025-02-20 15:12:25.992","lastEventTime":"2025-02-20 15:12:27.346","sessionEventsCount":7}
My concern is the second window created where it is "correctly" created, but what I would actually expected is that. 1. startTime and endTime that accordingly just describe the window time, should be different from the first window 2. the second and the third window should be "merged" since the second one should have started a new window that events of the third one would have fallen into I'm trying to figure out correctly how to properly understand what I'm doing wrong. Any help would be very great gratitude thank you
p
Hey there Antonio! Let me check if I understood it correctly. I'll just rephrase with my understanding and see if I'm not missing anything. You want to create slots of time processing and you want to always purge the data of the previous event. Is this right? So you are wanting to implement a session window and this
TriggerResult
is probably controlling something outside of this scope. So how is this
TriggerResult
handled later on? I'm wondering if having a
keyBy
doing this event wouldnt help. Because having different keys (you could add your window tuble to your key) would make sure Flink is in control of segmenting the stored data and you could rely on TTL. Perhaps I'm missing something but happy to iterate over it 😄
a
Hello @Pedro Mázala 🙂 I'm trying to explain myself better. I have events incoming (belongs to user events). I need to create session. A session starts as soon an event for a user arrive and it last until there's a gap of fixed amount of time in between last message received for this user. For doing so I'm using
EventTimeSessionWindows.withGap
Copy code
KeyedStreamBigquerySnowplow, String eventsGroupedByUserID = eventsSource.keyBy(BigquerySnowplow::getClientId);
Copy code
DataStreamBigQueryEventsSession sessions = eventsGroupedByUserID
                .window(EventTimeSessionWindows.withGap(Duration.ofSeconds(7)))
Now this works as intended, but I need to add a custom trigger on top, because I don't want this window to grows indefinitely. I want this window to close after a fix amount of time no matter what since first event for this user arrive Something like
TimeGapTrigger.of(Duration.ofSeconds(10))
Copy code
DataStreamBigQueryEventsSession sessions = eventsGroupedByUserID
                .window(EventTimeSessionWindows.withGap(Duration.ofSeconds(7)))
                .trigger(TimeGapTrigger.of(Duration.ofSeconds(10)))
This Trigger is a Custom Trigger that I'm trying to implement, that saves the very first event
timestamp
encountered for this user, and then check if
gapDuration
passed, and in case close the window.
p
Got it. I think the missing bit may be this. I'll edit your trigger code and show what I think would work. Non tested code
Copy code
@Override
    public TriggerResult onElement(BigquerySnowplow element, long timestamp, W window, TriggerContext ctx) throws Exception {
        ValueState<Long> firstEventTimestamp = ctx.getPartitionedState(descriptor);

        if (firstEventTimestamp.value() == null) {
            firstEventTimestamp.update(timestamp);
            ctx.registerEventTimeTimer(timestamp + maxSessionLengthMillis);
        }
        
        /*
         * We do NOT fire the window right away here because
         * we only want to fire after the maximum session length passes,
         * or if the inactivity gap triggers a session window close
         * at the operator level.
         */
        return TriggerResult.CONTINUE;
    }

    /**
     * Called when an event-time timer you've registered fires.
     */
    @Override
    public TriggerResult onEventTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {

        // If an event-time timer fires, it implies that the (firstEventTs + maxSessionLength) has been reached.
        // We want to fire AND purge the window immediately.
        return TriggerResult.FIRE_AND_PURGE;
    }

    /**
     * If you don't need any special handling on processing-time timers,
     * you can simply ignore or do nothing here.
     */
    @Override
    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) {
        return TriggerResult.CONTINUE;
    }
I think the issue was on closing the session on the new event insted on the timer
a
@Pedro Mázala I had as well the
onEventTime
method overwritten as you mentioned for closing the window. Just to clarification I'm running the job with the following Parameter Window:
EventTimeSessionWindows.withGap(Duration.ofSeconds(7)
CustomTrigger:
TimeGapTrigger.of(Duration.ofSeconds(10))
Removing the
FIRE_AND_PURGE
on the
onElement
methods gives me now the following output:
Copy code
WINDOW1: {
    "startTime": "2025-02-20 15:12:06.065",
    "endTime": "2025-02-20 15:12:25.605",
    "firstEventTime": "2025-02-20 15:12:06.065",
    "lastEventTime": "2025-02-20 15:12:18.605",
    "sessionEventsCount": 11
}

WINDOW3: {
    "startTime": "2025-02-20 15:12:25.992",
    "endTime": "2025-02-20 15:12:34.346",
    "firstEventTime": "2025-02-20 15:12:25.992",
    "lastEventTime": "2025-02-20 15:12:27.346",
    "sessionEventsCount": 7
}
What we can notice is that that compared to the previous results we don't have anymore the WINDOW2
Copy code
{
  "startTime": "2025-02-20 15:12:06.065",
  "endTime": "2025-02-20 15:12:25.605",
  "firstEventTime": "2025-02-20 15:12:17.099",
  "lastEventTime": "2025-02-20 15:12:17.099",
  "sessionEventsCount": 1
}
Because this event felt in the WINDOW1. But taking a look at it, we can see
Copy code
{
  "startTime": "2025-02-20 15:12:06.065",
  "endTime": "2025-02-20 15:12:25.605",
  "firstEventTime": "2025-02-20 15:12:06.065",
  "lastEventTime": "2025-02-20 15:12:18.605",
  "sessionEventsCount": 11
}
What I need to have is that
startTime
should be
2025-02-20 15:12:06.065
(as it is) but
endTime
should be
2025-02-20 15:12:16.605
(compare to 151225), This is because what I'm trying to achieve with my custom trigger is that a window should be closed after 10 seconds (
TimeGapTrigger.of(Duration.ofSeconds(10))
) from startTime. But it doesn't. It actually close only when there are no more message after 7 seconds (
EventTimeSessionWindows.withGap(Duration.ofSeconds(7)
) since last message received.
In the previous output the Window2
Copy code
{
  "startTime": "2025-02-20 15:12:06.065",
  "endTime": "2025-02-20 15:12:25.605",
  "firstEventTime": "2025-02-20 15:12:17.099",
  "lastEventTime": "2025-02-20 15:12:17.099",
  "sessionEventsCount": 1
}
contains exactly the only records at 151217 that I have for this user, after the 10 fixed seconds of closing windows I put in my trigger. Because the window start at 151206, 10 seconds passed, it closed the windows because that's what I want, and it should create a new one (like it does). But once again the
startTime
and
endTime
for this new window created is wrong. Plus the Window3 should be merged with Window2 because Window2 is now the "leader" of this specific timespan (151217-151227)
p
So what you actually want to do is a system timer? Because the timer will trigger based on your watermark. If you want it to be strict 10 seconds I'd suggest you do it with system time rtather than event time. But this may bring some other pains. Happy to chat over it 10m if you want to 😄
a
That would help indeed 🙂