We are currently designing the next iteration of Nextflow's module system: workflow modules Please feel free to have a look and comment. Currently it's looking like we may diverge a bit from nf-core's subworkflow system, so it would be good for folks to check our reasoning here

Are you not worried about multiplying the modules if each workflow can have their own local modules?

Yeah maybe having

<module>/<process>/<version>/

and referencing those specific versions would prevent code duplication here while still being able to use different versions.

I think that will interfere with includes. You would have to specify the version in the include path 😬

Ah I thought you could use multiple versions of the same module natively. I must have remembered that wrong

how does this work with subworkflows that require a config to work?

ah, I didn't know nf-core was doing subworkflow configs. I thought they decided not to do this for modules so I assumed the same was true for workflows https://github.com/nf-core/modules/blob/master/subworkflows/nf-core/quantify_rsem/nextflow.config

Meh, personally I don't like it

There are a few subworkflows that don't work if you don't set some kind of options

Yeah, but you know about it

If it were a module I would say that these defaults should just be in the process definition. But I don't know if these subworkflow defaults also happen to be sensible module defaults

Some thoughts: • Vendoring their own dependencies is how subworkflows should have always worked - GOOD • Terminology here is now horrific -

modules

vs

workflows

vs

subworkflows

vs

processes

it’s all so jumbled. Maybe it is too late to pivot but I would move from

modules

->

components

as the encapsulating term and then just have

process/module

(pick ONE) vs

workflow

as the distinction within that •

subworkflow

configs are real and necessary IMO. I will continue to fight on the fact that the flexibility of setting via config is one of the huge draws of nextflow over some other workflow managers. However given that subworkflow config should be setting code that needs to generally be fixed values to make the modules work as intended when running together (e.g. maybe a reads data pipeline that is specifically designed to handle CRAM would need to set the relevant flags on several modules) there is some scope to find a way to concretize those values in the subworkflow itself I think. • To be of any real use over

nf-core/subworkflows

this will need to support cross namespace dependencies (i.e. a

mynamespace/custom_subwf

should be able to depend on

nf-core/fastqc

and

otherorg/customproc

assuming they are both in the registry too

Regarding the terminology, I think we are converging on this: a module is a standalone process or workflow. I plan to avoid "subworkflows" entirely because they are just workflows But we need modules as a distinct concept to represent reusability outside the context of a single pipeline (module registry, module CLI,

nextflow module run

vs

nextflow run

)

Given that

nf-core/modules

is probably the most widely used existing context of the term

module

I predict it will be a source of confusion. However if that choice is locked in on the nextflow side then I think we should in fact consider renaming the

nf-core

terminology to avoid having one term refer to two subtly different concepts,

We could potentially use

nf-core/components

for the repo name...

IMO having to configure defaults for a process in a config makes development harder and more fragmented than it should be. I fully agree that it should still be possible for pipeline users to configure processes that way but I would not complain if it would also be possible to set this all up in pipeline code so I don't have to go looking in multiple places to determine how a module works

There are two major issues with

subworkflow

configs as they currently stand: • they have to be manually wired in • even then they don’t work out-of-the-box for aliased subworkflows (related nf issue) • fragile if users can and do want to modify them The issue with trying to take some of that configuration into the workflow itself: • if it is e.g.

ext.args

then users may legitimately want to and be safely able to extend them and the workflow function as intended (think customising an optional report format ) • modules would likely need to be rewritten to take that kind of content as inputs There is the possibility to solve the second one of those by doing something dumb like injecting the required args into a meta map of the channel input and then documenting that users need to wire up their config but that is a horrifically janky and laborious trick.

Given that

nf-core/modules

is probably the most widely used existing context of the term

module

I predict it will be a source of confusion.

However if that choice is locked in on the nextflow side then I think we should in fact consider renaming the

nf-core

terminology to avoid having one term refer to two subtly different concepts,

Actually I think we are on track to better align Nextflow and nf-core on these terms For a long time Nextflow defined a "module" as "a script included by another script". With these recent developments we are now defining it specifically as a standalone process or workflow, much closer to nf-core's definition Based on the current ADR, nf-core would move all "subworkflows" into the

modules

directory and just call them "workflow modules" vs "process modules". Then the repo name

nf-core/modules

actually becomes more accurate

The ext.args are used in several different ways: • Making a very generic module work for the specific use case of the pipeline (e.g. changing the output format via a flag) • Setting parameters that have a default, which is not necessarily the one you want • Allowing people to override or change options from that which was set by the original pipeline developer

According to the definition of 'modules' (dictionary.cambridge.org/dictionary/english/module) it is a part of a set that together forms one whole. So in that definition it actually makes sense to see a standalone pluggable process on the same level as a standalone workflow. I get that this might be confusing but TBH it does make the most sense if you really think about it.

yeah, nf-core chose to equate process == module. Nextflow never did that

Currently they are dynamic per-sample, because you can define them in a closure. This is used as a closure in a bunch of modules at the moment, but generally because they need to be set to something associated with the

meta.id

from the previous process. e.g. bam or vcf files where they have inside them a value which needs to be selected on. See e.g. github.com/nf-core/sarek/blob/…/mutect2.config#…

// PAIR_VARIANT_CALLING
        withName: 'MUTECT2_PAIRED' {
            ext.args   = { params.ignore_soft_clipped_bases ?
                                "--dont-use-soft-clipped-bases true --f1r2-tar-gz ${task.ext.prefix}.f1r2.tar.gz --normal-sample ${meta.patient}_${meta.normal_id}" :
                                "--f1r2-tar-gz ${task.ext.prefix}.f1r2.tar.gz --normal-sample ${meta.patient}_${meta.normal_id}" }
        }

Now in this case, the

--normal-sample ${meta.patient}_${meta.normal_id}

could easily be incorporated into the module itself. And especially the optional output file

*.f1r2.tar.gz

should be being controlled by a boolean as to whether to generate it

If we want these settings to truly live with the module, we have to think about how to define them in the process and supply them through params. Otherwise users will have to manually copy+paste the ext config and fiddle with process selectors

to be clear, that Mutect2 example both of the args that are being set with the closure could easily be fixed by changing the module itself (improving I would say), with only a little functionality loss (where you are passing bam files that have been generated outside of sarek itself, and so the normal sample tags inside the bam file doesn't necessarily have the same structure as shown here)

Dynamic per-sample is often used where a pipeline takes multiple data types. Eg it might be used in a minimap2 process to set the mode based on whether your meta declares the inputs as ONT,PacBio,Illumina etc

To be of any real use over nf-core/subworkflows this will need to support cross namespace dependencies (i.e. a mynamespace/custom_subwf should be able to depend on nf-core/fastqc and otherorg/customproc assuming they are both in the registry too

You can already do that with nf-core https://nf-co.re/blog/2025/tools-3_3#installing-subworkflows-with-components-from-different-remotes @Arthur

I know, my point was more that unless NF implementation supports it then all the improvements/advantages of having subworkflows vendor their own deps (rather than going the nf-core route of it being flat) still won't justify using it in my opinion.