I really need your help I got this exception in my...
# troubleshooting
u
I really need your help I got this exception in my flink: IOException: The record exceeded the max size of a sort buffer. The current max is … How to configure flink to increase this max size to solve this error? Thanks!
l
Can you provide some details like: • Flink version • Batch or Streaming • Do you have keyed operations? Asking because of sorting by key when in batch mode. • If streaming, then what state backend do you use? • What is your current setting for the total taskmanager memory? • What is your current setting for
taskmanager.memory.managed.size
?
u
Hi, 1. Flink 1.16 2. Flink batch using datastream api 3. Yap. I have 2 streams, i am doing keyby and then keyedcoprocessfunction. The record that i am collecting in that keywdcoprocessfunction is too big 4. Hashmap state backend 5. The default setting - running locally
I get this error only in batch execution mode
n
How to configure flink to increase this max size to solve this error?
As was alluded to above, you need to increase the
taskmanager.memory.managed.size
value. Fine grained memory configuration is quite tricky as there are quite a few rules (ratios, min/max values) etc., so you might have to play around with it.
u
I did played around a lot with these configuration and non of these works. I get this exception only for batch exeuction mode, when i move back to stream execution mode it works
l
Something to remember as well is that if you run this in batch mode, then each set of events associated with a key group will be sorted and processed before the operator moves on to the next key group.
Also, in batch mode no state is kept between keys being processed, and the backend state isn't used. Flink will also buffer the data on the taskmanagers filesystem between operators processing the batched data.
The exception you shared - can you share the exception with the stacktrace as a snippet?
u
Thanks, I dont have the entire stack trace but this is the main error. I tried to move to stream exeuction mode (for the same code) and the exception solved. You have idea why i get this exception only in the BATCH mode? In addition, when i am using the stream exeuction mode my job is also batch (because all of my sources are bounded), so what batch exeuction mode even gives me? Its only nice too have? Thanks you @Lee Wallen!
l
You have idea why i get this exception only in the BATCH mode?
I don't know why, but it could be due to batch mode processing the pipeline differently than streaming mode.
In addition, when i am using the stream exeuction mode my job is also batch (because all of my sources are bounded)
The app isn't "batch" because of the sources being bounded. Setting the execution mode to batch execution mode makes the job run in batch mode. Stream execution mode can use both bounded and unbounded sources. Batch execution mode requires bounded sources. This is described here. The Execution Mode documentation also has links to FLIP-134: Batch execution for the DataStream API and FLIP-140: Introduce batch-style execution for bounded keyed streams, which explains in detail how the data is processed differently between execution modes.
what batch exeuction mode even gives me?
You can reprocess your data and always get the same results (valuable for data validation), and you can re-run job steps that have failed without having to run previous job steps. The Flink docs for Batch Shuffle mentions performance tuning and troubleshooting tips.