Hi , anyone tried airflow and use along with an ex...
# random
s
Hi , anyone tried airflow and use along with an existing flink cluster together? Any documentation around this?
f
How would you be using them together? Airflow just runs code on a schedule, so it should be quite trivial to launch flink jobs from it. You might want to look into deferrable operators if the job is long running.
s
Can be for flink Cluster provisioning and job submissions to be automated,.. should be able to submit and manage jobs from Apache Airflow.
f
You might need to write your own operator to make it work with your infrastructure, but if it can be defined in code then Airflow can run it on a schedule
πŸ‘ 1
m
I'm assuming that you're thinking about Airflow in a batch / bounded mode only, given that streaming / unbounded applications are never ending
s
Yes @Martijn Visser