hei! one question regarding taskmanagers and task slots. I have a flink cluster with task num slots = 6 which is defined because the job with maximum parallelism is 6 as well. Is that correct? I also understand that task managers spawn as long as new slots are needed. Is there a limit for that? A task manager is a JVM process but has some memory configured per task manager. Is this the bottleneck for spanning more task managers? Thanks!

parallelism should be your number of task slots per node x the number of nodes (i.e task managers). Task managers will increase as parallelism increases. you can set this with the

parallelism.default

configuration. Your bottle neck will depend on your workload. for example, if you have too many task managers you can spend a large portion of memory/time shuffling data between the nodes

ok thanks. But what is the limit of task managers you can have? Is this limited by the total memory of the node? currently we only have one node but 3 task managers so there is no correlation node with task managers. Is it safe to make the assumption that the resource manager will spawn as many task managers as needed while there are enough resources (memory in the node) available for them?

What do you mean when you say you have 1 node and three task managers? AFAIK a taskmanager IS a node, so I'm not sure I understand how you are using the term "node"

you are right, node for me is machine or EC2 instance. so we have 1 ec2 instance and 3 task managers

To answer your question then, a resource manager will try to scale to match the parallelism you set assuming you have resources available

fair enough. thanks. so, the limit are the resources e.g. memory / cpu right? as long as I have resources will spawn new task managers right?

Not necessarily. It sounds like in your case you are trying to provision taskmanagers on a single EC2 instance so in that case its limited by that EC2 instances resources. In other examples, task managers can span across multiple EC2 instances and therefore are not limited on these resources