Hi , anyone tried airflow and use along with an existing flink cluster together? Any documentation around this?
f
Fredrik
03/22/2023, 11:19 AM
How would you be using them together? Airflow just runs code on a schedule, so it should be quite trivial to launch flink jobs from it. You might want to look into deferrable operators if the job is long running.
s
sairam yeturi
03/22/2023, 11:35 AM
Can be for flink Cluster provisioning and job submissions to be automated,.. should be able to submit and manage jobs from Apache Airflow.
f
Fredrik
03/22/2023, 11:52 AM
You might need to write your own operator to make it work with your infrastructure, but if it can be defined in code then Airflow can run it on a schedule
π 1
m
Martijn Visser
03/22/2023, 2:52 PM
I'm assuming that you're thinking about Airflow in a batch / bounded mode only, given that streaming / unbounded applications are never ending