Hello community, We have Flink operator deployed a...
# troubleshooting
b
Hello community, We have Flink operator deployed and are trying to configure HPA for TaskManagers using FlinkDeployments. But it is not clear for me from the documentation if it is supported.
c
Okay, you are one step ahead of me. Really good question on HPA. I will need the same. Have you tried it yet? In theory task managers are just pods, so if you have metrics working in the cluster it would work. Have you given it a try yet, or are you looking for example YAML?
Hey Bianca, I hope this helps. I was able to get HPA working. Here's is my HPA yaml. In order to have the measurements to scale based upon, you'll need to add the resources to your TaskManager Pod. apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: flink-cluster-hpa namespace: flink spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: flink-taskmanager minReplicas: 1 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 60 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 90
Make sure your POD spec for that task manager has these resources: requests: cpu: "500m" memory: "1Gi" limits: cpu: "2000m" memory: "2Gi"
b
This looks nice! Im testing it rn
I try to use it but it does not work given that im using FlinkDeployment object in place of deployments. so when I get the deployments within the namespace there is only a "Deployment"object for the jobmanager but not for the taskmanager. are you using the FlinkOperator? when you deploy are you using FlinkDeployment objects to deploy the app?
I need for it to scale up the amount of TaskManagers based on the memory usage and been trying (with no sucess) to use the following example : https://github.com/apache/flink-kubernetes-operator/blob/main/examples/hpa/basic-hpa.yaml Also discover that even setting up the amount of replicas on taskmanager for more than 1 does not work either .