Hi, I am trying to understand how Window functions work in Flink. To give some context, I have this pipeline which streams changes from Postgres (CDC) to Iceberg. Each record is part of a group, identified by

group_id

. I would like to compute some statistic over the group, and it to the record that is being streamed to Iceberg. If I understand the documentation correctly (not sure?), there is different approaches to computing statistics on windows. They, however, all depend on a slice of data being available (either defined by a time interval or a row count interval) to compute the aggregations. Would there be a way to ensure that the calculations are computed over the entire group? Thanks! 🙂

My guess is that you're interested in a global window? https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/#global-windows (And not a tumble, sliding/hop, cumulative or session window)

I am not sure if that will solve my issue. If I understand it’s documentation correctly, one needs to implement a trigger which determines at which point the results are aggregated? However, in my use case I actually need to fetch for a given group member all its other members to process the group as a whole. So basically, when an update happens to one of it’s members I need to fetch the rest of the group as well and process it. I was actually thinking of performing a self-join to fetch all related records to the group, but I am not sure if this gives any performance issues? Any ideas?

I'm not sure what you then mean what "fetch"; that sounds like you want to grab data from a source and then do an aggregation, but with Flink CDC the results are continuously being streaming in. Can you give an example of the question that you're trying to get answered?

Yeah, let me try to explain: • I have a Postgres instance having a table

docs

. This table have at least two columns

doc_id

, and

group_id

. So basically multiple docs can belong to the same group. • I have an external process updating my

docs

table. I plan to forward these updates to two other databases; Apache Iceberg and Elasticsearch. For that I use CDC. Updates to the

docs

table both include updates and/or deletes of existing records, as well as inserts of new records. • For my use case it is important to process and data as a group of

docs

. This basically means that if one of the

docs

get updates in Postgres, the pipeline needs to update the groups of docs in the downstream databases (Iceberg and Elasticsearch). • So basically I need to trigger an update of the group in downstream databases based on the change in the

docs

table. Hope this makes it a bit more clear?

I’ll try to look into your question tomorrow