Hi everyone. I am attempting to test the <new 1.1...
# troubleshooting
c
Hi everyone. I am attempting to test the new 1.19.3 release to take advantage of this enhancement: • [FLINK-33977] - Adaptive scheduler may not minimize the number of TMs during downscaling However, I'm not seeing a change in behavior. After a scale down event, the number of tasks decreases, but I'm not seeing task managers getting released like they should be. The number of Available Task Slots that is much greater than taskmanager.numberOfTaskSlots. With Flink 1.19.3, I would have expected the slot assigner to assign the remaining tasks to slots on the task managers with the highest utilization, so that the under utilized task managers could be released. This should cause the number of Available Task Slots to drop below the value of numberOfTaskSlots. I'm not seeing the Task Managers being released. Is there something that I'm missing? • jobmanager.scheduler = adaptive • jobmanager.adaptive-scheduler.prefer-minimal-taskmanagers = true • taskmanager.numberOfTaskSlots = 4 • Task Managers = 54 • Total Task Slots = 216 • Available Task Slots = 55
m
Exactly the same issue on downscale we have with 1.20.2 version. Using K8S Flink operator v1.12.1. And yes, it started from upgrade to Flink with adaptive scaling. No entire cluster restarts on scale came with downscale issue.
c
Hey @Maxim Duhovniy, We ended up seeing different behavior between 1.19.3 and 1.20.2. We did not see the same issue when running 1.20.2. Task managers were being removed as expected on scale down events. We are currently running version
1.12.1.b284
of the operator.
m
Great @Chris Harm , very useful info about versions. BTW, I see that FLINK-33977 declares 1.19.3 also as a fixed. In our case we did a mistake in deployment parameters definitions. We configured jobmanager.adaptive-scheduler.prefer-minimal-taskmanagers = false (it was true by default). I’m going to try with true this week. Will update you here.
Problem was solved after proper configuration described above, using Flink 1.20.2 Downscale works perfect now