Hello :wave: Question on Serialization/Deserializ...
# troubleshooting
j
Hello 👋 Question on Serialization/Deserialization in Flink Message Processing. Assume a job topology that uses 2 TaskManagers with 4 slots each. Looking for an answer with an official documentation link 😉 What is the expected behavior of serialization/deserialization in the following scenarios? 1. Within the same subtask ◦ When a record is passed between chained operators (e.g.,
Source → Map
). Does serialization/deserialization occur? 2. Between slots within the same TaskManager â—¦ If two subtasks are running in different slots of the same TaskManager, does Flink serialize/deserialize the record when passing data between them? 3. Between slots across different TaskManagers â—¦ When data is exchanged between subtasks (e.g., after
keyBy
) located in different TaskManagers, does Flink always perform serialization/deserialization during transfer? - I believe this is obviously yes, as it is different from JVM communication (Context: I am dealing with some performance fine-tuning, so I would like to know the behaviour to understand my job bottleneck)
d
1. No 2. Yes 3. Yes Note that in the case of operator chaining (1), either the object being emitted, or a copy of it, is passed to the downstream operator. Whether or not copying occurs is controlled by the object reuse configuration option. I'm pretty sure that these details are not discussed in the documentation. But the way it works is that if two subtasks are not chained, then the records are serialized into network buffers, which are then either communicated in memory, or via TCP. https://flink.apache.org/2019/06/05/a-deep-dive-into-flinks-network-stack/

Improving throughput and latency with Flink's network stack ...â–¾

YouTube · Flink Forward▾

Jul 11, 2019â–¾

j
Thanks, David!