Hey, in my team we're discussing streamGraph and s...
# random
d
Hey, in my team we're discussing streamGraph and serialization compatibility, my understanding is that the serialized version of the streamGraph is only used to: • spread the streamGraph over the network between flink components at runtime when deploying the job (e.g. jobmanager -> taskmanager) • stored in the H/A state for recoveries in these two scenarios flink does guarantee that the serialized version of the streamGraph is always generated and read by the same job jar (my understanding is the job jar should always be taken from the HA storage during recoveries, and from the blob store during job deployment)
c
The stream graph is never sent to task managers, nor is a stream graph ever serialized anywhere.
Are you referring to the JobGraph?
Because the JobGraph is stored in HA. But that is also not sent to task managers; it is only used within the JobManager.
Once the job graph is in HA we never regenerate it from the jar.
d
I probably mean the JobGraph indeed, when you write for instance a custom process function this one has to be serializable, its serialized form is what I'm interested about, flink makes sure that in both scenarios (deploy and recoveries) this serialized form is deserialized against the same jar that serialized it?
c
There is a SHA-1 checksum check to ensure the jar integrity. It doesn't ensure the jar wasn't modified in some way while preserving the checksum.
But usually (== in the absence of an external actor modifying the stored jars) it will use the same jar.
d
thanks for confirming!