There is something that keeps coming up when I discuss workflow managers with data science people which come more from the ML/AI and imaging analysis - and data eng - most of them never heard of nextflow. Argo, airflow, dataflow, prefect, are tech that keep being mentioned and no matter how much I advocate for nextflow I had little luck changing their minds. My question is: is there any big difference between the above and what nextlow can do or can't? Bceuase I think nextflow is superior in most aspects but I might be missing something? Any resources in comparing NF with those products?

I don't have any material, but some opinions: Nextflow was born in the bioinformatics space and therefore supports CLI tools very well. Almost all tools still expect to be operating in a filesystem environment and nextflow supports that neatly. With some exceptions, nextflow was designed to be run as one-off, large batches of samples. The other tools that you mention, I would mostly call data orchestrators. Their origin is in supporting ETL. They are mostly Python-based which is the language that most data engineers will be familiar with. Their focus is to make integration easy with as many data sources (and destinations) as possible and usually have no direct need for filesystem access at all. They are intended to be operated 24/7 with wide support for various schedules, triggers, and sensors to automatically start tasks.

Thank you. That's a great overview!

Paolo, the creator of Nextflow, maintains a pretty comprehensive list of workflow managers and pipelining tools. As you can see, there are plenty. Some are more popular than others, but it is generally a very heterogenous field. The reason why Nextflow became so popular in bioinformatics can imho be attributed to two main reasons: • Container runtime support : Nextflow early on adopted containers and allowed running pipelines inside them. Particularly for bioinformatics, where a pipeline needs to run a variety of different tools often written in different languages, this is a very important feature. The typical ETL pipeline will mostly interact with APIs e.g. from databases/data lakes/data warehouses and some kind of processing, monitoring and visualization infrastructure to derive business intelligence insights - a lot if not all of it can be handled inside the Python ecosystem or Python SDKs of external tools. • nf-core: In 2019/2020 I tried to found a start-up. My main audience with Nucleotidy were wet lab biologists and genetic counsellors and I wanted to provide them with user-friendly, yet flexible tools & mini-pipelines running entirely in the browser for generating biological meaning out of already aligned data, variant calls or count tables. Hence, there was no heavy lifting involved, and my eventual tech stack was build around software compiled to Webassembly, WebGL for data visualization and the pythonic Dagster for the workflows. However, regardless where I pitched this concept praising all the nifty technical details, I was met with a complete lack of interest. I approached investors thinking that the more bleeding edge and visionary my technology would be, the better. I stand corrected. They unequivocally told me, that they will not even attempt to assess the technical merits of my product, which would be futile anyway - all that mattered to them was, how many active users there are and how big the existing community around this technology is. I understood, that from a business perspective, the originality of your product is much less important than your ability to build a community. Frankly, I dislike Nextflow till today. Having no Java / Groovy background, there are a lot of things that seem not very intuitive to me and a few years back, when I first evaluated it, there were also not yet many conventions how to build “good” workflows with it. DSL1 was a blank canvas, and one could build very elegant or hideous workflows with it. But thanks to some early adopters among academic institutions (my current employer, NGI in Stockholm, was among them) and a bunch of very committed people who started and fostered the nf-core community together, this changed. One can still go completely freestyle in Nextflow, but a lot of consideration went into the sensible standards imposed by the nf-core pipelines and their canonical ways of doing things. Furthermore, the community created tooling, thousands of readily available, freely usable modules and provided thousands of hours of free support. Just do a quick estimate how much money a company would have needed to invest to create similar goodies around their tech stack while paying bioinformaticians a competitive salary... so yes, in my opinion it is predominantly the nf-core community that turned Nextflow into a mature workflow language.

I think this broadly answers my question as to why nextflow hasn't made significant inroads in ML and etl. At least for etl the needs are different.

you could ask in #C08H9MWG06T / #C07525632C9 / #C07MD07ULR2 as well. All pipelines or working groups in the areas you mentioned on their experience.

Good tip @Rike Hanssen! I wasn't sure which channel this question would fit. To be clear this is more about my own curiosity :)

for sure, I think this is a good channel for the discussion 🙂 I was more thinking that you might get some additional insights from converts and their daily challenges

One key reason is that machine learning and AI pipelines tend to be significantly simpler than those found in bioinformatics. Often, they involve just a single script—or a handful at most—and if you were to represent the pipeline as a DAG, it would usually be quite straightforward, especially if each node simply corresponds to a script. Switching to Nextflow would require rethinking the entire workflow structure, modularizing code across multiple files, and investing time to learn a new DSL that, while powerful, isn’t YET the most beginner-friendly. And in return, you’d unlock a plethora of sophisticated features… the vast majority of which you’d probably never need 😄.

But it could be prototyped locally and deployed wherever! Though seriously, I take your point about learning a new DSL. Heavy ask.

We would need to have dearly wished resources tailored to what they do to make it worthwhile. The company behind DVC has lots of experimentation dashboards and tools, for example, for benchmarking model trainings and so on. To get something you really want, learning a tool or a DSL is easy cake

But it could be prototyped locally and deployed wherever!

That is a feature that is far from unique for Nextflow. For ML applications specifically, you will also want to deploy or fine-tune/retrain your models on GPU infrastructure, which is arguably easier on platforms like HuggingFace or with Cog container deployments rather than generic Docker ones.

I found prefect/airflow/dagster/luigi: • struggle with files, as mentioned above • limited in how dynamic they can be. They require a lot of templating to handle lots of different variables (e.g. sample ID) • every concurrency or parallel operation has to be coded in, but every step involves this • required extensive boiler plate to wrap a CLI tool • sometimes, the deployment model didn’t make sense for me. E.g. they require client side software, or only launch onto one big node. These could all be dealt with, but at some point you’re re-writing the workflow engine inside another tool. Of course when using them for their intended purpose, none of these issues apply. Launching Nextflow with prefect/airflow is pretty common.