Hey! We use Flink to receive data from IoT sensors depending on the rate of the data that we receive, we want to generate events for connecting and disconnecting sensors i.e. if there is no data within 5 minutes, we indicate that the sensor is disabled there are two solutions to do this, use session windows or a timers which will be more efficient in terms of performance?

Performance-wise a custom ProcessFunction with timers/state adjusted to your needs might always be the better option. Windows might need to store more content that is actually required to perform the idle checking.

Thanks @Timo Walther! By the way, that’s exactly what we did. custom ProcessFunction with timers/state

Great. One additional "pro tip": make sure that you don't register too many timers, i.e. not one timer per incoming event but maybe a smarter algorithm? Maybe it is possible to allow a range min detection time/max detection time, this could allow you "batch" together timers. Event 1 sets a timer, and if event 2 comes right after 1 you don't need a fresh timer if it still falls into this range.

thanks for the tip 🙏 we pretty much did just that. but even with one timer per sensor, we are afraid that this will be a performance problem. since it is planned that we will have several hundred thousand sensors

If you keyBy sensor id before, you are parallelizing the stream well-enough for scalability. Several hundred thousand is a big number of course, for this experiments will be necessary, but in general Flink should be able to handle that.