Hello Flink dev I was not sure where to put these queries ! Since its more related to codebase and codeflow thought to put it here I was reading the Flink core code and doing play around using ThreadDumps to understand the flow. Few doubts i have didn't find proper answers over internet . This questions are purely based on my understanding and time spent on codebase. so understanding can be improper / incomplete It would be really great to know these answers from creators 😄 1. As per my understanding user job code get translated to

StreamGraph

->

JobGraph

->

ExecutionGraph

-> Physical Execution Why we need Stream Graph ? Why not Client make a job graph and directly submit ? What is the advantage of having StreamGraph ? 2. During Checkpointing , Flink is using Chandy–Lamport algorithm which is efficient because Jobmanager inserts checkpoint-Identifier in the record stream , whenever any operator running TaskManager (taskSlots) encounters this special record it will go in pause state will call the sleeping Thread responsible for checkpointing. This increases performance and entire processing flow doesn't go in halt But JobGraph has

IntermediateDataSet

and in physical Execution it has

ResultPartition

which wraps

ResultSubPartition

which holds output from operator. How this is handled ? When job is recovered from checkpoint, we can replay data from some specific point but how do we retrieve intermediateData created by operators in the DAG ? 3. Flink has top level SCALA api's for scala developers , scala is a language built on top of java 8 . Why we need

StreamingExecutionEnviroment

class and other implementation in SCALA also , looking into the code SCALA imports java classes itself only , felt like some wrapper in SCALA form on top of Java classes . What are the benefits it serve having these top level wrappers ? If there is really strong requirement and real use case , i see scala api's don't have all the implementation , felt like its incomplete . Why is it so? 4. When there is a flink job which needs to return some output to client why it uses

Accumulator

. TaskManagers write the output to accumulator then Jobmanager / Jobmaster reads and then passes it to client . What significance it provides ? Why not taskmanagers directly provide output to the client . this can reduse the workload on jobmanager , I think hadoop allows client result to be delivered by workers (not sure completely). What was the reason to choose this design in flink ? I am sure there would be good reason to choose this flow in flink trying to dig