Hi I was wondering how flink would execute these two flows i Apache Flink #random

Hi I was wondering how flink would execute these ...

Adesh Dsilva

09/15/2022, 2:30 PM

Hi I was wondering how flink would execute these two flows in a single app:

Copy code

sourceA -> mapFunctionA1 -> mapFunctionX1 -> mapFunctionX2 -> mapFunctionA2 -> sinkA -> sinkB
sourceB -> mapFunctionX1 -> mapFunctionX2 -> sinkB

As you can see X1 and X2 are common in these two flows so I am wondering, will Flink share the tasks across them such that events coming from both sources are processed by same group of tasks or will they be treated differently by Flink and events will flow into different versions of X1 and X2 for source A and B?

Chesnay Schepler

09/15/2022, 2:38 PM

X1 and X2 will not be shared.

Chesnay Schepler

09/15/2022, 2:39 PM

If anything the only shared thing is sinkB, IF you tell flink to union the streams before it.

Chesnay Schepler

09/15/2022, 2:40 PM

Mind you that you could tell Flink to share X1 and X2, by unioning the input streams and splitting the output again. But it's more hassle than its worth I imagine.

Adesh Dsilva

09/15/2022, 3:27 PM

Ohh, Got it. Thank you.

Adesh Dsilva

09/16/2022, 8:45 AM

A follow of question on this if you could help.. If I set execution environment parallelism or system parallelism as 10. Will both sources and their respective map/sink functions take 10 each or will they split it as 5 each since there are two streams? I am guessing everyone will take 10 irrespective of how many events are coming to source A or B unless I explicitly set them for each myself?

Chesnay Schepler

09/16/2022, 8:46 AM

I am guessing everyone will take 10 irrespective of how many events are coming to source A or B unless I explicitly set them for each myself?

Correct!

Adesh Dsilva

09/16/2022, 8:46 AM

Got it. Thanks!

2 Views

Open in Slack

Previous Next