Felix Terkhorn
04/06/2023, 3:05 PMTimer
capabilities in such a way: (if our project is successful) we’ll have potentially billions of individual states that may need to use processElement
/ onTimer
to schedule events into the future. Potentially the far future — like, a year into the future. Is Flink considered reliable for this type of long-term, delayed processing? Running a proof of concept with delays of a few seconds was super easy and successful, and on my local machine I pushed a ton of events through.
But testing these sorts of very long term delays isn’t feasible, so we’re really interested to hear feedback from the community about when (in time), and if, such a Timer
usage would begin to break down. How does Flink perform when scheduling things over multiple days & months?
Are there additional best practices and considerations that we should adhere to if we’re using Timer
s that are dated for execution far into the future?
Are there any concerns around prioritization of the `Timer`s firing at their due date, in the case where individual jobs experience high levels of traffic & contention?
Thanks very much for taking a look at my question!