Hello community,
We have Flink operator deployed and are trying to configure HPA for TaskManagers using FlinkDeployments.
But it is not clear for me from the documentation if it is supported.
c
Chad Theis
02/21/2025, 6:55 PM
Okay, you are one step ahead of me. Really good question on HPA. I will need the same. Have you tried it yet? In theory task managers are just pods, so if you have metrics working in the cluster it would work. Have you given it a try yet, or are you looking for example YAML?
Chad Theis
02/21/2025, 10:15 PM
Hey Bianca,
I hope this helps. I was able to get HPA working. Here's is my HPA yaml. In order to have the measurements to scale based upon, you'll need to add the resources to your TaskManager Pod.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: flink-cluster-hpa
namespace: flink
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: flink-taskmanager
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 90
Chad Theis
02/21/2025, 10:17 PM
Make sure your POD spec for that task manager has these
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "2Gi"
b
Bianca Falcone
02/26/2025, 4:13 PM
This looks nice!
Im testing it rn
Bianca Falcone
02/26/2025, 4:45 PM
I try to use it but it does not work given that im using FlinkDeployment object in place of deployments.
so when I get the deployments within the namespace there is only a "Deployment"object for the jobmanager but not for the taskmanager.
are you using the FlinkOperator? when you deploy are you using FlinkDeployment objects to deploy the app?