Hi everyone. My colleague raised an interesting question about usage of process substitution for concatenation tasks rather than using CAT_FASTQ , to avoid storing the large intermediate files. I don't see that pattern used much over in nf-core/modules, apart from one module. Are there any downsides to this? In my specific case most of the tools in the pipeline require the concatenated input, so I'm not sure if I should be worried about the potential time and I/O cost of process substitution in each tools process, versus the storage cost of using CAT_FASTQ once for all tools.

Yes there are downsides, which is why it’s rare in nf-core/modules: - A pipe isn’t seekable. Process substitution only works for single-pass streaming tools. Anything that indexes, seeks, or does two passes breaks on a FIFO. Your own link shows the tax: yak gets the reads twice — <(cat $reads) <(cat $reads) — because it can’t rewind a stream. - Repeated work. The concat runs inside every process, so N tools = N re-reads (and re-downloads) of the raw inputs, vs CAT_FASTQ paying it once. - Silent failures. Bash doesn’t surface the exit code of the substituted command — a truncated cat/zcat can pass unnoticed and your tool runs on partial data. pipefail won’t catch it. - Resume. CAT_FASTQ is cached; inline concat re-runs on every -resume. For your case — most tools need the concatenated input I’d use CAT_FASTQ once. It’s cheap (cat-ing .gz is a valid gzip), stored once (symlinked locally), and you can let Nextflow clean it up after downstream tasks consume it. Reserve process substitution for the single-consumer, guaranteed-streaming case. 🙂