As discussed at the maintainers meeting on Friday - I have made a port of a subworkflow to use the samtools/multicommand module. This (BAM_MARKDUPLICATES_SAMTOOLS) was the only subworkflow I could quickly find that chains multiple samtools calls on one another, and in this case the subworkflow can be completely replaced with a single module call. One thing I did note when putting this together with a real-world example, is that the samtools modules are not particularly consistent in their interface. For example, I discovered that

samtools collate

will write to stdout when you pass the

-O

flag, rather than just passing

-

as with many of the subcommands, which required some bugfixes to the code I had already written. So I think it's possible a module like this could hold quite a few bugs if not constructed with care, and well-tested so that at the very least each subcommand is tested as the first and last command of a pipeline, which might be onerous. And if we replicated this for other tools, they might require their own carefully-constructed carve-outs. Not sure that the above is necessarily reason to not have these modules, but it means they would definitely require some care in the way they are written so that the logic is clear and are not a mess of ifs and opt-outs. github.com/nf-core/modules/pull/11550/

Might be worthwhile to make a variant of this? github.com/nf-core/modules/…/sormadup

That's functionally the module that runs in BAM_MARKDUPLICATES_SAMTOOLS - collate -> fixmate -> sort -> markdup

I don't think that the module needs to be 100% bug proof before we make it available. It is via use that these really surface, and this should fail completely with that sort of error. It is also likely to be used by more experienced developers in the first place

Agreed - I bring it up more as a general caveat to implementing this kind of module - testing should be extensive to catch as many edge-cases as possible and where possible the code should be organised to avoid spaghetti by catching exceptions in a clean way

ok, so

multicommand

? We agree on this for the name?

Works for me. Alternative would be

pipedcommand

if we want to be more explicit that the module creates a chain of piped commands, but I don't know if we care enough

All modules and subworkflows currently give all

task.ext.cpus

cpus to each sub-command, letting the OS / execution engine manage the number of CPUs being used.

And that's how this module works also, currently, but we discussed in the maintainer's meeting last week if we would need to think about the allocation of CPUs among these samtools commands

I agree with capping the number of CPUs given to some commands. For instance,

view

probably can’t print things faster with 12 CPUs than with 4 CPUs (or less) – stdout is likely the bottleneck. But not trying to keep the sum of all threads given to all commands equal to

task.ext.cpus

. This would lead to an underuse of the CPUs

I suppose this also depends on whether something like

view

is the final command. If I'm filtering to a 100GB+ CRAM file, I might want the spare threads at the final stage for compression.