Bringing up this one again, now that I'm back from leave: github.com/nf-core/modules/pull/11550 In the previous discussion (https://nfcore.slack.com/archives/C043UU89KKQ/p1777986494144269) folk were cautiously in favour of modules like this, at least pending some update to the module guidelines. I think modules like this can be valuable particularly in terms of preserving disk space by streaming operations, rather than storing intermediates on disk through successive module calls. Does anyone has some thoughts on this example implementation? I have tried to restrict the options to those you might reasonably expect to want to pipe (there's no reason to support

samtools stats

, for example). If folk are generally positive and we can figure out roughly how the module guidelines should change to accommodate this, I'll write up an RFC 👍

Oh, we're back at this

I actually don't want this for samtools, I want it for pairtools. I currently have a local module "PAIRTOOLS_PARSESORTFILTER" which I would love to throw in the bin 😂

oh I see

I would love to be able to improve the resource efficiency of some streaming commands but I think this is something that needs to be handled quite carefully because I think it hands a huge footgun to a community which is often prone to shooting themselves in the foot. I’d also wonder if, given the complexity, it’s better left until after migration to record types begins. I also think it risks creating myriad duplicate but subtly different implementations of what people are already doing. I would propose that if such modules were adopted, then where they can replace functionality of current subworkflows, we should have some way of distributing the “correct” implementation of the streaming module to replicate the subworkflow. I think this is worth being cognizant of as Ben has designed the current

nextflow module

setup around the idea that modules do not distribute e.g. config and I think this would possibly challenge that idea

That's a fair point about replacing subworkflows - I hadn't really considered that we currently have any that would be completely deprecated by an approach by this, but a cursory glance does show that we have some. So a migration strategy would definitely be useful, though I don't necessarily have a problem with both co-existing for a long time.

I’d also wonder if, given the complexity, it’s better left until after migration to record types begins.

My suspicion would be that if this requires changes to the guidelines, this is not going to happen any time soon. But I don't necessarily see how migrating to records would be any more or less complicated for these modules, if we want them.

I think this is worth being cognizant of as Ben has designed the current

nextflow module

setup around the idea that modules do not distribute e.g. config and I think this would possibly challenge that idea

I haven't followed the discussion about

nextflow module

too much, could you elaborate a bit?

Well basically, subworkflows sometimes distribute config, when this was raised in a thread about expanding the module command to cover subworkflows, it seems that distributing config was not really considered. I'm not sure if the subworkflow replacements a chaining module would allow would be modules with config or subworkflows, but if it's the former they may not be distributable

I've wanted this for so long, but I agree with what Arthur said.

I almost pinged you @Mahesh Binzer-Panchal, I remember you wanted that