Hi Was wondering if there is a tool or a combination of tools which anybody has used to act as a coordinator for jobs submitted in session mode Looking for mainly one thing 1. Scale up and scale down taskmanager pods based on some kind of queue of jobs 2. Additionally management of jobs in case of failure of job manager Afaik operator automatically does this part but wanted to confirm Thanks

Looks like you need an orchestrator? Have you checked airflow? It even supports to submit a Flink job - https://airflow.apache.org/docs/apache-airflow-providers-apache-flink/stable/operators.html

Yes I did @Jaehyeon Kim But I didn't find a lot of documentation for scale up , scale down logic which is the main logical piece I need

Let say your Flink session cluster is deployed in the standaone mode on k8s, you'll have k8s Deployment for the job manager and task managers - https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#session-cluster-resource-definitions If you have an airflow job that checks CPU/RAM metrics periodically, you'd be able to adjust the replica count of the task manager? In this scenario, it'll be easier to add more task managers. I'm not sure which would be a good practice to remove task managers. Normally an existing job will fail temporarily and restarted in an available task manager. If it is acceptable, you can just reduce the replica count. Otherwise, all jobs should be stopped with savepoint and restarted later, which would be quite complicated.

Yea I think out of all the options I explored the scaling down one was the trickiest issue I had explored keda for some queue based scaling but yeah all issues boiled down to existing tasks getting removed I think I'll stick with either a hybrid between session and application mode or just application mode based jobs A follow up is there a dashboard in the existing operator where I can view running jobs History server shows completed jobs only and it won't be feasible to port forward each jobmanager in application mode