I'm curious what are the largest deployments of flink you have seen in terms of the number of TM. I have a use case where I am joining 2 streams using KeyedCoProcessFunction. Source streams have 60k-100k messages per second and my early estimates are that I will need parallelism of at least 1000. Planning to use HashMapState since my state is not that large and I want to have as high of a throughput as possible. Is this level of parallelism concerning? I'm especially worried about checkpointing given that HashMapState does not support unaligned & incremental checkpoints

This is one of the largest that is public https://www.alibabacloud.com/blog/four-billion-records-per-second-what-is-behind-alibaba-double-11---flink-stream-batch-unification-practice-during-double-11-for-the-very-first-time_596962

I believe Unaligned Checkpoints does work with

HashMapStateBackend

. Checkpoint alignment actually has little to do with the state backend.

thanks for sharing, article is pretty high level, but what I got out of it is that 4 billion records per 35,000 jobs -> ~114k/s per job 1,500,000 CPU per 35,000 job -> ~42 CPU per job On average ofcourse. Curious to hear whether folks have deployed a single stateful flink application with 1000's of TMs

You don’t have to assign a full CPU to a job. The Alibaba deployment is more then 1000 TMs. There are other companies as well, but that information isn’t public

Based on the throughput I was able to achieve so far with running up to 480 pods being insufficient

I can see that it can be challenging indeed. It’s one of the things multiple companies (incl mine) include in their product offering

That being said, I'm still pretty early in the tuning process so its possible there are other bottlenecks but I was curious to hear whether there are some theoretical limits in terms of TM that I'm not aware of

And now on Flink as well 😉

Might be relevant: https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics I was able to perform a join of 9 Kafka topics with 13 TB state, required ~150 beefy taskmanagers.

To improve the performance of a job doing joins, watermark alignment and buffer debloating can help a lot.

For anyone stumbling on this thread, I noticed huge throughput increase after enabling buffer debloating AND setting

taskmanager.network.memory.floating-buffers-per-gate

equal to parallelism

Nice!