Hello, I’m considering a few possibilities for a Flink job, and would like to ask for insights from the community 🙏 There may have been gotchas that I’ve missed out or things I’m yet to be aware of. Let’s say I have one event per min on a Kafka topic. I need to do some operations on the current element and an element from a certain time offset, X min ago. Thinking about 2 options: 1. 2 consumers, one reading from latest offset, another reading from latest-X offset. Only the leading consumer will help move the event time on the operator. Cons: a. If we need to run the same logic for various offsets (5min, 1440min, 10080min, etc), there would be a fan-out issue (1 new consumer needed for each time offset). Perhaps we can split the topic into multiple smaller topics before applying the same logic on them. b. Anything else that we need to be aware of if the offset is big? e.g. 24 hours? 7 days? Theoretically, as long as the retention time for the Kafka topic exceeds this, it seems fine…? c. Perhaps can consider having 2 kafka topics (replicated), if there’s issues with the above. 2. Use Statefun to send delayed messages (non-blocking, fault tolerant) One single consumer, with 2 downstream operators, one operator process it as is, another operator sends delayed events to the first operator. Question: a. The same logic can be achieved by using a process function and storing all the elements within the time offset in a ListState (can get big if time offset is big); think ‘pseudo sliding window’. But instead of having a big ListState, perhaps it’s more optimised/efficient when having a separate operator deal with the delayed events? b. If we need to run the same logic for various offsets (5min, 1440min, 10080min etc, can be on separate threads, parallelism), would the Statefun delayed messages approach be scalable? How does it fare/how does it work compared to Timers? (I’m not familiar with Statefun and would like to know what to watch out for. For timers, we can try RocksDB backend if it’s too much I guess). Thank you very much 🙂

I am not sure why an event a minute warrants using Kafka and Flink. That seems overkill, but that's not the question. Kafka reads from its queue and you can control what offset you start at but you'll be reading all messages in each stream as fast as you can, so your option1 doesn't seem viable. You probably want to use sliding windows, with a 1min slide, and window sizes matching the different offsets you need to go back to, then use a window process function to take first and last events in the window and do what you need to do with them. Otherwise, you can keep a fifo style map state in a process function to keep the history with event time as a key, cleaning up processed events and adding new events and be able to pick any timestampnback in the history. You just need to keep as many events as the largest offset you will process.

Thank you Emmanuel 🙂 The throughput upstream is much higher, it’s just that after some processing we expect 1 per min at the latter stages. Sliding window off-the-shelf didn’t work after a certain point as the number of windows opened at any one time would be large given certain cases. I guess we’ll have to test it out on option1, process function, and maybe look into statefun more.