I'm looking at using BATCH mode. The documentation states that data will be sorted and handled key-by-key (paraphrasing). I'm wondering where Flink stores the data in the mean time? is it all in memory? I looked through the docs for information on memory requirements, but didn't find anything.

Hi @Ron Cohen - See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution_mode/ where it references https://cwiki.apache.org/confluence/display/FLINK/FLIP-140%3A+Introduce+batch-style+execution+for+bounded+keyed+streams#FLIP140:Intr[…]oposedChanges. The key section (though without a lot of details) is:

We will introduce a sorting step (with potential spilling, reusing the UnilateralSortMerger implementation) before every keyed operator for sorting/grouping inputs by their keys. This will allow us to process records in per-key groups, which will enable us to use a simplified implementation of a StateBackend that is not organized in key groups and only ever keeps values for a single key.

Thanks for the pointers Ken!

Hey @Ron Cohen In the

BATCH

mode Flink will try to spill data if necessary while sorting. It will use the direct memory budget for in-memory operations and will start spilling to disk if it reaches a memory threshold as configured by

taskmanager.runtime.sort-spilling-threshold

. You can also control the maximum number of files it may try to create. Lastly it uses the temporary directories as configured in

io.tmp.dirs

thanks for elaborating Dawid!

Sorting works on data input of a keyed operator. An operator receives data for keys in a key group it got assigned. An operator usually processes multiple keys. Sorting always happens on that level within the assigned key groups. If I understand your question correctly then the sorting will happen for a key across all partitions.

gotcha, yes. Perfect. Thanks again!