Hi all Would like to get a feature into nf-core tools, afaik it isnt possible at present without manual installs (subworkflow with commit hash, then dependencies with commit hashes, one after the other) Currently,

nf-core tools subworkflow install

will install all module dependencies using meta.yml precise commits can be fed to

nf-core tools module</subworkflow> install

, but (i believe) there is no option for adding a precise commit to a meta.yml as follows, such that its dependencies are also pinned (although perhaps the commit state of modules are pinned to the parent subworkflow via the command line install when a precise commit is provided?):

components:
  - samtools/sort:
      git_remote: "<https://github.com/nf-core/modules.git>"
      branch: master
      git_sha: "abc123..."

In practice, this means that as nf-core modules change, a subworkflow can be installed on two different dates - the subworkflow code may be identical but the modules that nf-core tools installs as dependencies can differ. While I appreciate that changes to a module in nf-core trigger test cascades for subworkflows which use said modules, this doesn't really handle the underlying issue. This version control issue has been a bit of a headache for my team for some time and I thought it was time to bring it up here and potentially look into making a PR. This would probably touch

nf-core lint

as well. Anyone have any thoughts on this? would appreciate it.

This makes a lot of sense to me. Sort of in the adjacent line, I have been thinking for a while on how we keep subworkflows in sync. The upgrade burden on the person changing a module can be quiet high when they then need to bump a number of subworkflows as well. I am guessing this could have side effects we don't really account for now, like having multiple versions of the same module installed, because different subworkflows require different module versions

Yeah unfortunately I don't think a GitHub rep allows versioned modules in a particularly organic way like other dependency libraries would, part of the cost of optimising for open source contribution friendly framework currently utilised

The real answer in my opinion is that - absent a proper versioning and a dependency resolver (and maybe even still then) - subworkflows design is wrong. Subworkflows are actually just workflows which we reuse. So many of their issues could be resolved if they were treated that way and packaged their own dependencies beneath them. I suggested this a long time ago but for various reasons it wasn't seen as a workable idea. Without a real dependency resolver (and dependency resolution is not trivial - just look at

pip

) adding version pinning to individual subworkflow meta yml is a bad idea because what do you do when two subworkflows pin different - potentially incompatible - versions? Consider additionally "patched" modules. I know lots of people are a fan but if you have a very common module (something like

samtools

) that you patch for some use case in your pipeline this could easily affect the way a subworkflow running that module operates in your pipeline. TL;DR: the problem is the way subworkflows are designed currently. Within the current design I don't think there is a good or safe solution to pinning dependency versions.

Yeah, I advocated for copying all modules into pipelines under a subworkflow with nf-core #CE5LG7WMB many moons ago for this reason, but I was outvoted

So here's a potentially strange idea. What about using Pixi? Is it viable? How much could be offloaded here? They've demonstrated you can package manage agent skills with Pixi: https://github.com/pavelzw/pixi-skills. Why couldn't modules and subworkflows be handled in this way too? Native lock file, and possibly more.

Using an environment manager just seems potentially overkill to me. I think that decisions need to be made about: • patching • can 1 module exist in different versions in 1 pipeline? Without clear guidance on what a pinned or versioning logic looks like for those things you can’t really start to decide upon how the package manager should work. For example if in theory if you say: • a module can exist in 1 version per workflow/subworkflow - 1 installation per “workflow/subworkflow” dir basically. • where version ranges are used (not pinned versions) pipelines try to have the fewest different number of versions possible • patch conflicts are not handled by any package manager and are up to the workflow devs Then you can have a lot of flexibility to allow pipelines to mix and match versions where necessary but also to attempt harmonisation of versions. I strongly believe that the actual tool is somewhat arbitrary and that what you need is a clear and unambiguous spec that simply allows developers to write tools that implement that spec in whatever fashion/language they choose. The new nextflow module functionality somewhat formalises a module spec for the language (based out of the nf-core module spec), but subworkflows (or re-usable/installable workflows) are a much thornier issue.