Hello folks. First time using slack, but I do real...
# troubleshooting
e
Hello folks. First time using slack, but I do really need a hand here. In the DataStream API I’m using session windows. Sometimes a window has already fired and emitted downstream, but then a late event arrives that causes this window to merge into another session window. I’d like to run custom logic at the time of the merge (for example, emit a retraction or compensation event for the earlier window that already fired). I know that Trigger.onMerge exists, but it does not allow me to directly hook into the fact that “window A that had already fired is now being absorbed into window B” and use it. To get more specific - in my use case, I would like to send to a side output the id of the events that existed in the merged window, so I could delete it from the data base and consider it irrelevant 👉 Is there a recommended way in Flink’s DataStream API to implement a custom callback on window merge so I can handle these “already-fired-but-now-merged” windows?
a
Have a look at the
MergingWindowAssigner
and
EventTimeSessionWindows
and see if you can make your roll your own implementation of that and a custom
TimeWindow
. I imagine you'd be able to add some "meta" or perhaps a UUID for new windows that is carried over to the new merge result window when a merge happens. This might enable you to unique on the receivers end? Eg.. Here's the skeleton to illustrate my idea:
Copy code
public class UUIDTimeWindow extends TimeWindow {

    private final String uuid;

    // for new windows
    public UUIDTimeWindow(long start, long end) {
        super(start, end);
        this.uuid = UUID.randomUUID().toString();
    }

    // used when merging windows
    public UUIDTimeWindow(long start, long end, String uuid) {
        super(start, end);
        this.uuid = uuid;
    }

    public String getUuid() {
        return uuid;
    }
}
Copy code
public class EventTimeSessionUUIDWindows extends MergingWindowAssigner<Object, UUIDTimeWindow> {

    @Override
    public void mergeWindows(Collection<UUIDTimeWindow> windows, MergeCallback<UUIDTimeWindow> callback) {
        // look at EventTimeSessionWindows and TimeWindow.mergeWindows for implementation
        // make sure the first window's UUID is copied to the result window
    }
And at a later stage in your pipeline, after the windowing has happened and before writing to your sink, you'd need to enrich the payloads with the window UUID. Use a
ProcessWindowFunction
to get access to the `UUIDTimeWindow`:
Copy code
public class UUIDWindowProcessFunction<IN, OUT, KEY> extends ProcessWindowFunction<IN, OUT, KEY, UUIDTimeWindow> {
    @Override
    public void process(KEY key, ProcessWindowFunction<IN, OUT, KEY, UUIDTimeWindow>.Context context, Iterable<IN> elements, Collector<OUT> out) throws Exception {
        var uuid = context.window().getUuid();
        // add uuid to your OUT object
    }
}
I don't know if this is the simplest route, but I reckon it would work (provided you can de-dupe/unique/compensate on the receiver end)
e
Using this method makes the codebase larger and causes a lot of code duplications... And also - in my head, I would like to access the state of the windows that got involved in merge (the process function state for example), and using this APIs it's simply impossible. I'm trying to solve this problem differently, by using a process function with string state of "fired sequence id" that is changed on the first call to the event's sequence id, and a custom clear function that checks if this state's value exists and if so sends it to a side output of "already fired but now needs to get deleted" sequences. The problem is to differentiate clear calls of a merge event, and clear calls of actual window that just got closed I'm hoping I successfully explained my problem..
a
In the method I described you can add what ever window state you'd like to the UUIDTimeWindow. In my example I only added a UUID, but if you need other state you can easily add that, and access it later on using a
ProcessWindowFunction
. The
ProcessWindowFunction
also provide access to each record along with it
Window
. Be mindfull how you use the
ProcessWindowFunction
though: if added to the stream via a call to
.window()
then function requires that all data in the window is buffered until the window is evaluated, as the function provides no means of incremental aggregation. In other words you'll receive the full contents of the window in
ProcessWindowFunction.process()
. If you instead add the
ProcessWindowFunction
using a call to, for instance,
.reduce()
then elements are not buffered and you instead received the reduced record along with it's window in
ProcessWindowFunction.process()
Anyhow, perhaps I'm misunderstanding the problem and your particular context. The solution you describe I'm not fully understanding. Perhaps a code example would help me understand the intention.