Hi all! I am trying to understand if usage of windows are relevant for me. My goal is to sync a postgres database (source) with Elasticsearch (sink) by making transformations on the data.I have the following conceptual problem: • source A: CDC • source B: CDC • source: C: CDC • table D :

D = select * from A left join B on <http://a.pk|a.pk>=<http://b.fk|b.fk> left join C on <http://a.pk|a.pk>=<http://c.fk|c.fk>

• table E:

E = select <some aggregations> from D group by D.group_key

• table F:

F = select *, <some expensive transformations> from E

• F gets inserted in Elasticsearch. So what I notice is that because of the left joins, and some events arriving later than others, table F get’s updated a couple of times for the same key. Which is logical I think. However, since the transformations made to create F are expensive, it recomputes that over and over again. For me it would be better if there was some sort of timed buffer on which events to E could arrive such that the expensive transformations do not need to be calculated on every event updating E. I am trying to understand if windows can help me out here, but I am unsure. Also, it is very important that the aggregations in E do contain all of the group elements. Thanks!

Did you try minibatching? https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/tuning/#minibatch-aggregation

This looks interesting yes! I will have a look at it.

btw: I have a similar issue, https://apache-flink.slack.com/archives/C03JKTFFX0S/p1679443155741349, currently I'm struggling to setup minibatching to use event time instead of processing time. I'm considering using windows as well, but no luck so far.

Is your usecase also with CDC?