How would one achieve better performance for Flink...
# troubleshooting
i
How would one achieve better performance for Flink SQL joins to ensure data isn't being sent around from one instance to another for each join? Like, is there a way to ensure data affinity, meaning all the tables are consistently keyed and thus end up on same instance? Am I thinking about this wrong?
m
The partitioning for joins depends on the join keys. If you are joining orders with customers, for example, the partition would be on customerid. If you then join the result with products, the partition would be on product id and the intermediate result from the previous join would have to be sent to the worker node for the particular productid. That in and of itself is pretty efficient in Flink. Performance issues arise if those joins need to be maintained as inner joins in which case Flink has to shuffle around a lot of data on updates which is where most of the inefficiencies come from. A temporal join in this particular example would eliminate that (mostly).
i
What do you mean by shuffling it around? Wouldn't the index just stay on the partition where that join occurs?
m
If you do an inner join between an orders stream and customers table, and you update a field on a customer with id=5, then flink will produce updates for all the previously produced join records for orders with customer id 5 (i.e. where the join predicate matched). And then those updates are sent to downstream joins. Over time, that accumulates to lots of updates that need to be shuffled around the system. If the email update isn't relevant to subsequent processing, that would be a lot of wasted effort.
i
Ah, I see.