מייקי בר יעקב
03/11/2025, 6:55 PMLee Wallen
03/12/2025, 1:23 AMtaskmanager.memory.managed.size ?מייקי בר יעקב
03/12/2025, 6:34 AMמייקי בר יעקב
03/12/2025, 7:54 AMNicholas Erasmus
03/12/2025, 10:10 AMHow to configure flink to increase this max size to solve this error?As was alluded to above, you need to increase the
taskmanager.memory.managed.size value.
Fine grained memory configuration is quite tricky as there are quite a few rules (ratios, min/max values) etc., so you might have to play around with it.מייקי בר יעקב
03/12/2025, 9:19 PMLee Wallen
03/20/2025, 5:54 PMLee Wallen
03/20/2025, 5:58 PMLee Wallen
03/20/2025, 6:02 PMמייקי בר יעקב
03/20/2025, 6:22 PMLee Wallen
03/20/2025, 6:57 PMYou have idea why i get this exception only in the BATCH mode?I don't know why, but it could be due to batch mode processing the pipeline differently than streaming mode.
In addition, when i am using the stream exeuction mode my job is also batch (because all of my sources are bounded)The app isn't "batch" because of the sources being bounded. Setting the execution mode to batch execution mode makes the job run in batch mode. Stream execution mode can use both bounded and unbounded sources. Batch execution mode requires bounded sources. This is described here. The Execution Mode documentation also has links to FLIP-134: Batch execution for the DataStream API and FLIP-140: Introduce batch-style execution for bounded keyed streams, which explains in detail how the data is processed differently between execution modes.
what batch exeuction mode even gives me?You can reprocess your data and always get the same results (valuable for data validation), and you can re-run job steps that have failed without having to run previous job steps. The Flink docs for Batch Shuffle mentions performance tuning and troubleshooting tips.