Hi all, if its ok I'm going to cross post this her...
# meta-omics
n
Hi all, if its ok I'm going to cross post this here, since this is a frequent situation in our meta-omics world. Any thoughts on process substitution vs separate processes for concatenation? https://nfcore.slack.com/archives/CE6SDBX2A/p1785765681829679 Would be applicable for any of the tools that support merging files from the same sample across sequencing runs.
j
IMO disk-space issues are a much bigger concern with Nextflow than the potential for a small additional overhead using process substitution. When I was running nf-core/mag on a small cluster, I regularly ran out of disk space as the work directory ballooned. I think we should in general at nf-core be really careful about spilling to disk and try and keep things small, otherwise these pipelines will not be nearly as generally useable as they might promise with community effort/accessibility etc.. So I'm all for process substitution in general, or to use named pipes!
agree 1
We make use of them quite liberally in our own modules repo: github.com/search?q=repo%3Asanger-tol%2F…&type=code