Hey folks! Very basic design question as I'm start...
# troubleshooting
s
Hey folks! Very basic design question as I'm starting to learn about best practices using Flink (complete beginner to distributed data processing). Consider a scenario where you're tracking posts in a forum topic over time, and performing aggregation over the post data. You decide to select the top level
Forum ID
as your key for Flink to partition the data prior to running your aggregation logic across nodes. Each node maintains state over a time period, and outputs the aggregation results to a sink. (Happy path). Now, you realize that certain
Forum IDs
are turning into hot-keys, where the node responsible for them are processing a lot more data than the average node. What do you do in this scenario? An idea I was toying with in my mind was to append a partition suffix at the source, so that the data gets redistributed within Flink (ex:
forum-abcd#partition-1
,
forum-abcd#partition-2
, ...etc). While this distributes the aggregation logic across partitions, I would still need a way to collect the data back from all the different partitions and reaggregate them by the Forum ID (without the partition suffix). Is there native support for this in Flink? The other alternative I was thinking of, which seems overkill, is to chain another Flink cluster to ingest data from the first one and partition it by stripping out the suffix. This would enable the second cluster to perform aggregations over pre-aggregated data which vastly reduced the number of unique data points per `Forum ID`(as they have already been compacted during the first aggregation) and output to my final consumer workers. Would this be a valid approach?
p
Would you have any grainer data then the forum id?
s
That would be great if I did, I'm just making up this scenario to reason about how to handle hot keys. In the real world you probably won't have a forum with millions of topics being created per second, but I can't seem to come up with a good solution here (unless I have a more detailed/grainier partition key).