Hello everyone,
We are running a
Flink session cluster in standalone mode on Kubernetes, and we deploy
multiple TaskManager deployments under the same session cluster, each with different CPU/memory configurations.
This allows us to support both lightweight and CPU-heavy pipelines in the same Flink session.
Example setup:
• TM type A → low-spec (1 CPU, 4GB)
• TM type B → high-spec (4 CPU, 16GB)
• TM type C → mid-spec (2 CPU, 8GB)
All TaskManagers register with the same JobManager.
Our problem:
When we submit a job, we cannot control
which TaskManager type the job will use.
Flink may place CPU-heavy tasks on low-spec TMs because we cannot specify:
• TaskManager affinity
• Slot affinity
• TM selection rules
• Resource-class filtering (beyond resource profiles)
We previously used Native Kubernetes mode via the Flink K8s Operator, but it doesn’t allow different TM resource specs within the same session cluster.
Switching to standalone mode gave us flexibility to run multiple TM deployments, but we still cannot control
which TM the JobManager assigns tasks to.
We found a similar question from 2021:
https://stackoverflow.com/questions/68531175/task-manager-affinity-in-flink
Any guidance would be appreciated.
Thanks!