Hi team, I have some questions regarding Autoscaler, could anyone help me with them? Thanks! 1 - It seems for flink jobs using flink operator to realize autoscaling, the only option is to enable the Autoscaler feature, could you please confirm this? 2 - From the Autoscaler doc, we see

The autoscaler currently only works with the latest Flink 1.17 or after backporting the following fixes to your 1.15/1.16 Flink image

, and the updated flink operator version is 1.7. Our current flink version is 1.16.1, and the flink operator version is 1.5. • Is it possible for us to only upgrade the flink version to 1.17 without upgrading flink version to 1.7 (or 1.6)? • If it's possible, what configurations we need to do to enable Autoscaler on flink operator version 1.5? 3 - Could you please provide a list of metrics automatically observed by Autoscaler? • Will it include CPU load, memory, throughput and kafka consumer lag? • Is there any configurations related to kafka consumer lag that we can setup to scale job by making Autoscaler monitor it, like some threshold?

You might find this post more useful than the docs https://flink.apache.org/2023/02/27/apache-flink-kubernetes-operator-1.4.0-release-announcement/ We will work on improving the documentation soon :)

@Gyula Fóra Thanks for this link! I have similar questions about the metrics being used by the operator autoscaler to make scaling decisions. • The documentation (and the above blog post) mentions "target processing rate" but never defines it. What does "target" mean here? If there a way to configure target processing rate for various vertices in the graph? • Is there any way for autoscaler to also take into account system metrics (CPU/memory util) to make the scaling decisions? From documentation it seems like it takes into account "busy time per second", but that would also include the time flink is waiting on IO or on external dependencies (like reads/write to DB/network), and including IO wait time in the busy time to make scaling decision wont be ideal.

Hi @Gyula Fóra, thanks a lot for the reply! I have checked the release post you have provided, the charts are helpful in understanding how flink operator reacts to the load increase/decrease. But I am still confused on something: • 1 - Which version of flink operator shall we upgrade to? Can we use flink version of 1.17 + flink operator version of 1.5? Is flink operator version of 1.6/1.7 required for Autoscaler? • 2 - It says

Source scaling requires the standard connector metrics, currently works best with Kafka sources

, does this mean Autoscaler will automatically monitor kafka consumer lag, and will scale up to prevent the lag accumulate? Can we set up some threshold/max limit for kafka consumer lag, once it's reached, the job will be scaled up immediately to decrease the lag? Thanks!!!

Lot of questions here, let me try to tackle a few: • You should always use the latest Flink Kubernetes Operator version (currently 1.5.0, 1.6.0 is being released next week), regardless of the Flink version of your jobs. The operator supports multiple flink versions at the same time, currently 1.13 - 1.18. However the atuoscaler functionality requires a Flink job version of at least 1.17.

Hi @Gyula Fóra Thanks so much for all the detailed and helpful info! Really appreciate your help! gratitude thank you

Thanks for the detailed help @Gyula Fóra ! Small followup regarding the busy time metric: If a vertex (specially a sink node) is writing to an external DB, and the DB is highly overloaded (and is the main bottleneck), that vertex will have close to 100% busy time. In such a case, scaling up based on the vertex busy time will make things worse. Hence the need to be able to scale up solely on CPU utilization rather than “flink vertex utilization” (i.e. % busy). Maybe I am misunderstanding something here?

You either use the operator autoscaler or reactive, you can't use both at the same time. I think the autoscaler is much more advanced than anything you can get with reactive mode

Thank you @Gyula Fóra @Vaibhav Gosain, I previously thought that for flink jobs using flink operator, it's impossible to use KEDA, cause TaskManager instances are created and destroyed by the Flink JobManager, TaskManager instances aren’t in a replication controller, so they can't be “scaled up” using traditional Kubernetes techniques like KEDA. And without KEDA, reactive mode itself cannot realize autoscaling. So I thought maybe for flink jobs using flink operator, Autoscaler is the only way to realize autoscaling. I am not sure if my understanding is correct, need some confirmation. @Gyula Fóra Thanks!!

Hi @Gyula Fóra, I was also exploring the autoscaler functionality of the

flink-operator

and am not able to figure out if the autoscaler will scale the TaskManager pods instead of just adjusting the job parallelism otherwise it will lead to over provisioning of resources on the TaskManager side. The other thing I wanted to ask is if its necessary to enable the

ADAPTIVE

scheduler in order to for the operator autoscaler to kick in? Thanks

I suggest trying it on minikube / local env with the included examples

I actually tried it on our dev EKS cluster after enabling the operator autoscaler and did not observe any change in parallelism/taskmanagers under load.

You don’t need adaptive scheduler but you need the Native deployment mode (default for the operator )

Thanks @Gyula Fóra. I will check the operator. I think I missed the part that operator autoscaler does not support standalone deployments.

forget the autoscaler, just think of how a simple parallelism setting works with native Kubernetes deployments. You set parallelism to X, the job starts with 0 TMs and the jobmanager will compute and create new TMs to have enough for X

Understood. Thanks for the explanation

Adaptive is not the same as reactive

Understood. Thanks a lot!