Hi all , question on unbounded aggregations in Flink: I need to compute all-time metrics (e.g. count distinct, leaderboards) rather than windowed ones. The straightforward approach is to keep everything in keyed state, but I'm worried about state growing unbounded pretty quickly. Is there an established pattern for this? One option I was considering was to use async I/O against our DB to check whether a key/combination already exists, and computing the distinct counts from there instead of holding it all in Flink state. (not ideal) Curious to hear if there's a better approach (approximate sketches, external store, something else). gratitude thank you

Yeah this is possible, we're doing quite a bit of this. On Flink 1.x the issue you're going to run into with large state in slow deployment times. With state in the 200GB range expect around 10-20 min depending on your configuration. During this time no data will be processed so no counts will be updated. Flink 2.x has really improved this situation and we're running jobs with 3-6 TB of state using async state access APIs. These jobs deploy in a few minutes. There are some tradeoffs here as well though, the big one being autoscaling parallelism is not really an option at this state store size. Increasing parallelism is fine but decreasing it will cause Flink to reshuffle your data leading to a very long deployment time.

Hi @Francis Altomare, first, sorry for the late reply! This is perfect, we’re going with the 2.x versions and have clear guidance around downscaling. I agree with your last point: keeping state small is key, rather than flushing full rows. We’ll keep that design in mind. Thanks so much for your help!