Hi the airflow integration in datahub is very cool...
# ingestion
s
Hi the airflow integration in datahub is very cool. I wonder if there is much interest/demand in similar integration with Lyft Flyte?
👍 1
l
We will look into this. What would the workflow be across Flyte and a tool like AWS sagemaker?
@high-hospital-85984 interested to hear your thoughts as well about how you use Flyte as opposed to Airflow
h
We’re going to deploy it for ML model development and training. Not too many learnings yet, but the use is of course more iterative: try this preprocessing method, tweak this parameter, and see if things improve. Also, there will be more parallel development branches as multiple data scientists work on different parts of the same model pipeline.