I had what one might consider either a moment of extreme inspiration or absolute insanity... Wondering what folk think of this sketch for a "samtools pipeline" module? Idea is that you provide a list of samtools subcommands (e.g.

[merge, view, sort, markdup]

) and it constructs a module script that streams a BAM or set of BAMs (dependent on starting tool) through all the subcommands, each of which pull in their respective

argsX

variable. Could be a nice way to easily perform a bunch of streaming operations though some tools while minimally impacting disk space (which is a problem if you otherwise chain a bunch of modules!) https://github.com/prototaxites/modules/blob/samtools-pipeline/modules/nf-core/samtools/pipeline/main.nf

A bit unorthodox, but I like the idea. It's a bit funny to implement shell pipelines inside Nextflow pipelines though.

Do you mean https://github.com/nf-core/modules/pull/4571/changes ?

I didn't think I'd be the first to propose such a thing 😅

That discussion seems to have gone stale long ago, but the points raised there likely still hold (e.g. how to make this module user-friendly and how to name it correctly)

I still think this is valuable

You might be in the ideological minority once again

I don't think the module would be particularly un-user-friendly... but also don't think every module we have has to have some level of simplicity/friendliness. Samtools is a suite of tools that is a-priori designed to be piped together to create pipelines, I think anyone reaching for a samtools module wouldn't be too surprised to find one that allows them to construct a sequence of commands. I would just say to be sure to have a good bunch of usage documentation in the

meta.yml

There are other tools where you might also reasonable expect similar modules could be constructed because they're built around the same principles -

bedtools

,

pairtools

come to mind. It's the latter I was thinking about when I built this - I just thought samtools would be a better start point for the proposition.

You might be in the ideological minority once again

Sure, but James is on holiday for 2 weeks, so if we just get on with it it'll be merged and he'll never know.

Don't forget about Mahesh, he was the main critic of the linked PR

I'll acquiesce on

ext.suffix

if we can have this

I thought that Mahesh was actually in favour of this in the first place

I thought that Mahesh was actually in favour of this in the first place

Not sure, @Mahesh Binzer-Panchal?

I would say that the caveat would be that they have to be pipelines for a single tool suite - i.e. every command should be a

samtools

command, we don't want complex multi-tool suites where you pipe samtools to bedtools to awk. And the tool suite should be designed for this kind of sequential piping in the first place.

☝🏻 that alone for me will be a reason not to open this can of worms canofworms

I think there is something to be said for the fact that single-tool pipeline modules are likely to be a lot more re-usable than "whatever arbitrary set of tools I want" modules would be. The latter are likely to be too use-case-specific in ways that would mean they would be rejected even if they weren't constructing arbitrary bash pipelines.

We already have one exception from the one-tool rule (clear performance/storage advantage), we could add intended subtool pipelines as a second one. Still an exception, not a free pass to make whatever people feel like.

Or a clarification that "clear performance/storage advantage" could pertain, for example, to pipeable tool suites

I have a feeling that this clause was added specifically for samtools pipe operations

My (untested!) sketch is here: https://github.com/prototaxites/modules/blob/samtools-pipeline/modules/nf-core/samtools/pipeline/main.nf You could add a bunch of guardrails to it as needed, and perhaps a nice comment above the script block that shows how an example command is constructed from an example input

Could you give a couple of invocation examples?

Yeah, exactly

@Phil Ewels, you are allowed to vote in the pool to give you opinion too 😉

I'm in favour of modules that improve resource efficiency. My objection was solely that the guidelines stated atomicity, which I was never really in favour of, and also implementation style (I prefer list + join to create strings rather than using + as I find it much easier to read and it's not creating and discarding as many objects under the hood). I've always faced the argument that these are too complicated for new-comers and it should be as easy to compose as possible, but that has always come at the cost of resource efficiency. Pipeline developers could always write more efficient modules themselves then. So to summarise, I want a place for multi command modules, guidelines, and appropriate docs. ( I don't have the time to spearhead it though )

Definitely in favour of more resource efficiency - I think it's one of the main places that limits the portability and general usability of nf-core pipelines at the moment. I'm happy to develop the sketch above a bit further if folk are happy to see a fully fleshed out and working example, but I'm starting paternity leave any day now so would also not be able to help out with extending/developing a set of guidelines for this type of module, at least for a few months

( congrats 🎉 - enjoy the time )

Ok, so that sounds like we have a cautious yes to this. I'd say the next step would be suggestion a modification to the guidelines that we can argue about 😉