I have a complex java flink challenge. I have a f...
# troubleshooting
u
I have a complex java flink challenge. I have a flink job that have 2 bounded source: A and B. For A and B i am assigning an H3 index and then perform keyby by this H3 key. Now i have 2 keyed stream so i am doing keyedcoprocessfunction that join A with list of Bs by the H3 key - until here everything is good. The output of this join is A with list of all the related Bs. After that i have a process function that takes the Bs and perform heavy algorithm on it. Currently i am facing a problem that the algorithm process function that becomes after the join (ints in the same task in the flink ui - operator fusion) is skewed. I have 80 tasks for example and only 2 are running. I think its because i have much more Bs then As - a lot of join tasks says that the got 10000 records for example but zero output. So curently i am facing a hard data skew for the algorithm. I tried to do a keyby after the join and before the algortihm by id of As - but now i was faced with another problem: Each B is 401x401 matrix of floats. A join result can be A with List of 1000 Bs for example - its a lost of 1000 matrices of 401x401. The problem is that when i shuffle the stream of the join result it took so much time to serilize the join result - hourse just for moving the data from the join task to the algorithm task. Help me find solution, i want to solve the data skew problem and having control on the distribution of the algorithm, but i am facing this so much heavy serialization issue. Thanks!
a
May be think about these points Re-keying or custom partitioning after the join to reduce data skew. Broadcasting the
A
stream
to ensure more even distribution of work. Optimizing serialization using custom serializers or a more efficient format (e.g., Kryo). Using the RocksDB state backend to optimize memory management for large matrices. Avoiding operator fusion to better control task distribution.
u
Ok i can to
rebalance
after the join and then solve the data skew, but the problem is that my job is stack because its trying to serialize and do network shuffle of the join result. The join result is list of 1000 big matrices, and this makes the job stuck... You have idea how to solve this stuck problem after the join? Thanks