This message contains interactive elements.
# random
i
This message contains interactive elements.
plusone 1
s
👋 , The autoscaler via the K8s operator implements this FLIP, that scales appropriate vertices of a job graph with requisite parallelism. Reactive mode, however, scales the entire job graph, that may not be ideal for complex jobs (see the ‘Rejected alternatives’ section in the FLIP linked above)
i
Thank you so much @Singh!
@Singh I have a follow up question, maybe you can quickly point me to the materials on how to clarify it. So I'm confused about how Flink distributes the operators in the complex pipelines, and correspondingly how it choose then ultimate number of task managers. Assume we have a simple graph A -> B -> C -> D, taskSlots = 2, and parallelism = 100. In this case it simply divides 100 / 2 and gets 50, each task manager running 2 instances of each operator. But what happens when operators have heterogeneous parallelism? Say A (20) -> B (50) -> C (200) -> D (30) I'm confused in this case what exactly "task slots" represent actually, because in the former case it's simply the number or parallel instances of each operator per task manager. And here it's the same? So does it mean that the number of task managers would be MAX(operator parallelism) / taskSlots? So, in this case, 200 / 2 = 100? Sorry if it's something obvious / explained in the docs. I didn't find it and it confuses me quite a bit. Thanks!