Hello Folks, I am trying to experiment with task slots of task managers and assigning optimal resources to them. I have tried two approaches and able achieve optimal configuration with which application processes without any back pressure for the workload we have.I would need your suggestion on which approach is better . Flink job consumes from 4 kafka topics and writes 4 kafka topics. It has keyed state (rocks db as state backend) which it refers to perform event deduplication. I am deploying this flink job in application mode using flink kubernetes operator 1. Task manager having multiple task slots. In this approach, each TM is having more than 1 task slots (setting it to 10 in our case) and assigning higher memory and higher cpu (cpu: 2). This way each task slots shares managed memory but one slot might consume all the heap memory sometimes. So there is not heap memory isolation. With this setup, its observed that we need to set higher parallelism (100-160) for the operators that take heavy load. 2. Each task manager having only one slot. In this approach, each TM is having only one slot and assigning 1 CPU and optimal memory for each slot. This way each task is guaranteed to have certain heap and managed memory. No heap memory sharing among task slots. With this setup, we observed that application processed fine with lesser parallelism compared to first approach. So it is about more parallel tasks vs more resources to each task with heap isolation. Any thoughts on these approaches?

One factor to consider, if you are sensitive to latency, is that larger TMs with many slots will have larger heaps and longer GC pauses. Someday we’ll get Flink up to Java 17 and this will no longer be an issue, but right now this may be something to consider. However, since you’re using RocksDB, maybe your job(s) don’t create a lot garbage.

Thanks David for your point. Yes it is sensitive to latency. Is my understanding correct about the approaches I mentioned? Which one would you recommend ?