Hi Team, I want to use elastic scaling, do you know the plan when elastic scaling reactive-mode will become a GA feature? (as it states it is MVP feature currently) https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/

I suggest checking out the Kubernetes operator autoscaler: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ It is not using the reactive mode but this is an actual component that can do true elastic scaling

Thanks @Gyula Fóra! let me take a look ~

While the reactive mode sounds simple on a high level it actually needs a lot of custom logic to make it work for scaling the pipeline to the load

I see, so assume that I monitor a metric like average CPU usage and when threshold is beyond 80% I add one more taskmanager node ( flink does not take care of adding the taskmanager, i add it manually ). Then in this case, the elastic scaling is based on available taskmanagers and so this feature is already supported, is that right?

Yes but I don't think it will work very well if you use it like that . CPU / memory monitoring will only work in very specific cases

oh i see, thanks! which metric do you think could be better to monitor wider spectrum of cases?

Thanks for the reference , let me take a look 🙂

This is very useful, thanks @Gyula Fóra

Good thread to understand about reactive mode

@Gyula Fóra Thanks for the comments. So autoscaler work for elastic scaling based on load? - meaning it can automatically add new taskmanagers when monitored metric is not reaching target? Thanks.