Elad
08/24/2025, 8:15 PMAndreas Bube
08/25/2025, 7:04 AMMergingWindowAssigner and EventTimeSessionWindows and see if you can make your roll your own implementation of that and a custom TimeWindow. I imagine you'd be able to add some "meta" or perhaps a UUID for new windows that is carried over to the new merge result window when a merge happens. This might enable you to unique on the receivers end?
Eg.. Here's the skeleton to illustrate my idea:
public class UUIDTimeWindow extends TimeWindow {
private final String uuid;
// for new windows
public UUIDTimeWindow(long start, long end) {
super(start, end);
this.uuid = UUID.randomUUID().toString();
}
// used when merging windows
public UUIDTimeWindow(long start, long end, String uuid) {
super(start, end);
this.uuid = uuid;
}
public String getUuid() {
return uuid;
}
}
public class EventTimeSessionUUIDWindows extends MergingWindowAssigner<Object, UUIDTimeWindow> {
@Override
public void mergeWindows(Collection<UUIDTimeWindow> windows, MergeCallback<UUIDTimeWindow> callback) {
// look at EventTimeSessionWindows and TimeWindow.mergeWindows for implementation
// make sure the first window's UUID is copied to the result window
}
And at a later stage in your pipeline, after the windowing has happened and before writing to your sink, you'd need to enrich the payloads with the window UUID. Use a ProcessWindowFunction to get access to the `UUIDTimeWindow`:
public class UUIDWindowProcessFunction<IN, OUT, KEY> extends ProcessWindowFunction<IN, OUT, KEY, UUIDTimeWindow> {
@Override
public void process(KEY key, ProcessWindowFunction<IN, OUT, KEY, UUIDTimeWindow>.Context context, Iterable<IN> elements, Collector<OUT> out) throws Exception {
var uuid = context.window().getUuid();
// add uuid to your OUT object
}
}
I don't know if this is the simplest route, but I reckon it would work (provided you can de-dupe/unique/compensate on the receiver end)Elad
08/26/2025, 10:40 PMAndreas Bube
08/28/2025, 6:52 AMProcessWindowFunction. The ProcessWindowFunction also provide access to each record along with it Window. Be mindfull how you use the ProcessWindowFunction though: if added to the stream via a call to .window() then function requires that all data in the window is buffered until the window is evaluated, as the function provides no means of incremental aggregation. In other words you'll receive the full contents of the window in ProcessWindowFunction.process(). If you instead add the ProcessWindowFunction using a call to, for instance, .reduce() then elements are not buffered and you instead received the reduced record along with it's window in ProcessWindowFunction.process()
Anyhow, perhaps I'm misunderstanding the problem and your particular context.
The solution you describe I'm not fully understanding. Perhaps a code example would help me understand the intention.