Hi all! I'm new to Flink, and I'm trying to understand how Flink autoscaling works on Kubernetes. I'm using aws kinesis connector, so I want a fixed number of task manager pods. I've deployed Flink using the Flink operator version 1.13.0 with Flink 1.20. Deploying using

FlinkDeployment

, defining

spec.jobManager.replicas

works as expected, but when I'm trying to specify

spec.taskManager.replicas

it seems to be ignored. Does anyone know how can I manually configure the number of task manager pods? Should I configure it using HPA?

I think this should work, maybe check

parallelism

value, https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.13/docs/custom-resource/reference/#taskmanagerspec docs does say

spec.taskManager.replicas

takes precedence.

This is what I can't understand. I have set replicas to

2

for example, and I have only 1 pod

maybe it does not have work

I tried several things and it just looks weird. I;ve tried without setting replicas, set parallesim to 4 for example with number of task slots to 1

weird, we both must be missing something

Ah ok, that makes sense. I guess it's supposed to be some combination of number of task slots + parllelism

from github copilot chat • If you set CoreOptions.DEFAULT_PARALLELISM = 100 and TaskManagerOptions.NUM_TASK_SLOTS = 2: ◦ Standalone mode: operator would compute TMs = ceil(100/2) = 50 and create a TM Deployment with 50 replicas (unless spec.taskManager.replicas is explicitly set). ◦ Native mode: operator will pass DEFAULT_PARALLELISM to Flink. Flink’s native deployer / resource manager / scheduler determines how many TMs are started; if you use the adaptive scheduler Flink can request more TMs as needed; if you change per-vertex overrides the operator may call rescale APIs to adjust running job parallelism.

Ok I figured it out! I'm using java code for the flink application and the example I used was using

env.setParallelism(1)

! I removed this and redployed and now everything works as expected

nice flink