We are consuming from two Kafka topics to produce a single stream of data. A 24 hours tumbling event time window is applied to this stream along with an aggregate function. At the end of the 24 hours, the result of the aggregate is upserted into a mysql db. The problem its that one of the topics has significantly less messages than the other (on the order of tens of thousands vs millions) and in the case of replaying all the data (we retain 7 days worth in our topic), the smaller topic causes the watermark to advance and the daily windows to fire, resulting in the data in the other topic to be late and thrown out.. Is there a way to consume from multiple topics at a rate than maintains event time order? Also, in general are any best practices we should be aware of or are there other things to consider when aggregating together multiple dissimilar data sources?

One way to consume from multiple topics at a rate that maintains event time order is to use a Union operator to merge the streams from the two topics, and then apply the window and aggregate functions on the resulting stream. This way, the watermark and event time ordering will be determined by the larger topic, which will prevent the smaller topic from advancing the watermark too quickly and discarding data from the other topic. sorces: • https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#union • https://medium.com/@knoldus/flink-union-operator-on-multiple-streams-740c05d33a82

@Zsombor Chikán what if the data sources are intended to be entirely different schemas. Does schema consistency just mean that the key and event time are similar? For instance, in this problem Matt and I cared about aggregating data like fuel used while a vehicle is driving but in a feature we will need to tackle soon, we have to aggregate that fuel data with data from deliveries to determine how much fuel is used per delivery. Our two sources would have similar fields of employeeId and eventTime in their events but the rest of the data would be completely different.

You can also look at https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-alignment-_beta_, which is meant to address this problem.

@Kyle Godfrey what I wrote was only my first things at the top of my mind, but we can iterate on that.. so am I understood correctly, you mean data sources have entirely different schemas?

I guess what we had hoped for was a system in which events of different types would be ingested from the various streams and affect a stateful object. Things like one data stream tells us that a delivery route was started, another data stream tells us that the vehicle started moving, and a final one sends events when the driver slams on their brakes. Each of the data streams represent entirely different things and are orthogonal to each other. It would then spit out metrics like the average amount driving that was done for each delivery and the average number of hard brakes per drive for a driver. This way we’d be able to introduce new data sources over time for new sets of reported metrics since we have roughly six systems to integrate and likely multiple individual data sources from each system. The main challenge we seem to be running into is a lack of documentation/understanding around these operators

Thank you all for the info

@Matt Weiss so the watermark alignment worked?