מייקי בר יעקב
03/27/2025, 9:41 PMAbhilash S.R
03/28/2025, 2:18 PMA stream to ensure more even distribution of work.
Optimizing serialization using custom serializers or a more efficient format (e.g., Kryo).
Using the RocksDB state backend to optimize memory management for large matrices.
Avoiding operator fusion to better control task distribution.מייקי בר יעקב
03/28/2025, 2:24 PMrebalance after the join and then solve the data skew, but the problem is that my job is stack because its trying to serialize and do network shuffle of the join result. The join result is list of 1000 big matrices, and this makes the job stuck...
You have idea how to solve this stuck problem after the join? Thanks