Hi everyone , I'm new to Flink and currently exploring it for a fraud detection use case. I had a couple of questions I’d appreciate guidance on: 1. Large time windows: we’re looking to compute aggregates over windows as large as 6–12 months, with an ingest rate of around 10K events/sec. How feasible is this in Flink in terms of state size and performance? Any tips on managing that scale? 2. Prewarming aggregates: is there a clean way to bootstrap state from historical data so new jobs or aggregates can start with warm state instead of empty? Any resources or real-world examples would be super helpful, thanks in advance

Hi, You should use flink to process data daily and write another job to aggregates data daily to monthly, yearly.

Take a look at Flink’s HybridSource. I think it was designed with this scenario in mind.

What Draco said about HybridSource. It will give you the ability to read bounded data before unbounded data. This is just perfect for historical (bounded) data. About the windowing, what Vu Le said is reasonable. I've run pipelines with 15 days windows and they worked fine. The tradeoff here is: the bigger your window, more open tracks you will have and more events being processed you will have. So this may lead you to a huge state or memory spike. Adopting "aggregation layers" would be a nice approach. Group it by days and then, downstream you aggregate those events i.e. I hope this helps 😄

I've build a similar system for transaction aggregate at MoMo. So Flink job process streaming event to calculate daily feature and write data to Hbase (rowkey = user_id#yyyymmdd, qualifier_column = feature_name, ex: total_daily_transaction) Then I develop another aggregate service when want to get total_weekly_transaction, total_monthly_transaction, so it's only read a few rows in HBase. This help to reduce cost, because I don't want FLink job have process a large datastream (months/year). This will lead to many issue with Flink state/memory like Pedro Mazala mentioned above.