Sam Chakrabarty
02/23/2025, 2:01 AMForum ID as your key for Flink to partition the data prior to running your aggregation logic across nodes. Each node maintains state over a time period, and outputs the aggregation results to a sink. (Happy path).
Now, you realize that certain Forum IDs are turning into hot-keys, where the node responsible for them are processing a lot more data than the average node. What do you do in this scenario?
An idea I was toying with in my mind was to append a partition suffix at the source, so that the data gets redistributed within Flink (ex: forum-abcd#partition-1, forum-abcd#partition-2, ...etc). While this distributes the aggregation logic across partitions, I would still need a way to collect the data back from all the different partitions and reaggregate them by the Forum ID (without the partition suffix). Is there native support for this in Flink?
The other alternative I was thinking of, which seems overkill, is to chain another Flink cluster to ingest data from the first one and partition it by stripping out the suffix. This would enable the second cluster to perform aggregations over pre-aggregated data which vastly reduced the number of unique data points per `Forum ID`(as they have already been compacted during the first aggregation) and output to my final consumer workers.
Would this be a valid approach?Pedro Mázala
02/24/2025, 9:40 AMSam Chakrabarty
02/24/2025, 11:33 PM