Hi all, imagine I have a datastream containing sensor readings. The readings arrive in order (w.r.g to the event time). I would like to process the values of the sensors and do some processing, e.g. I would like to do a kind of integration (length of distinances in time times difference in the readings). How would I achieve that? Would I do a sliding window with two elements repeating every element, or would I write a stateful map function which internally memorize the previous sensor readings? (If the later, how would I do that? Thanks, Dieter

Hi @Dieter - one comment…in general, you can’t assume that events arrive at a function is perfect time order. If you have a source parallelism > 1, or any keyBy operation upstream of your function, etc. then you typically can no longer make that assumption. This is where the concept of “max lateness” can come into play, where once the watermark is at time X, you assume you won’t get any events earlier than X - the max lateness.

Hi @Ken Krugler, thanks for the tip. If I get my time series data via Kafka and there the data was ordered by event time, how could I do something like I have described?

You could try with a sliding window, set the window size to be something like what you want for max lateness, and then only process the first N (up to the slide time) events when each window triggers. This talk describes how I’d do it in a low-level process function, once BinarySortedMultiMap is available: https://www.slideshare.net/FlinkForward/introducing-binarysortedmultimap-a-new-flink-state-primitive-to-boost-your-application-performance

@Dieter to do the integration, you only need two points at a time, right? If you've keyBy’ed the stream according to your need (sensorId?) then I'd just do a simple RichMap function which stores the latest data point and integrates it against the incoming point, then updates state with the latest point