Hi all, I wanted to ask about the seemingly quite common use of

2>|

before piping to log files in nf-core modules. See for example the number of hits when grepping for this construct together with

.log

, in the nf-core modules repo:

$ pwd
~/src/nfcore-modules/modules/nf-core
$ grep -nr "2>|.*\.log"
hifiasm/main.nf:70:        2>| >( tee ${prefix}.stderr.log >&2 )
filtlong/main.nf:31:        2>| >(tee ${prefix}.log >&2) \\
trimmomatic/main.nf:41:        $args 2>| >(tee ${prefix}_out.log >&2)
narfmap/align/main.nf:42:        2>| >(tee ${prefix}.narfmap.log >&2) \\
gt/gff3/main.nf:30:        2>| >(tee "${prefix}.error.log" >&2) \\
seqkit/rmdup/main.nf:39:        2>| >(tee ${prefix}.log >&2)
bowtie2/align/main.nf:63:        2>| >(tee ${prefix}.bowtie2.log >&2) \\
merfin/hist/main.nf:42:        2>| >( tee ${prefix}.hist.stderr.log )
faqcs/main.nf:39:            2>| >(tee ${prefix}.log >&2)
faqcs/main.nf:73:            2>| >(tee ${prefix}.log >&2)
melon/main.nf:36:        2>| >(tee ${prefix}.log >&2)
ragtag/patch/main.nf:59:        2>| >( tee ${prefix}.stderr.log >&2 ) \\
ragtag/scaffold/main.nf:54:        2>| >( tee ${prefix}.stderr.log >&2 ) \\
motus/profile/main.nf:41:        2>| >(tee ${prefix}.log >&2)
motus/preplong/main.nf:34:        2>| >(tee ${prefix}.log >&2)
irescue/main.nf:39:        $args 2>| >(tee -a ${prefix}/irescue.log >&2)
dragmap/align/main.nf:49:        2>| >(tee ${prefix}.dragmap.log >&2) \\
rseqc/junctionannotation/main.nf:36:        2>| >(grep -v 'E::idx_find_and_load' | tee ${prefix}.junction_annotation.log >&2)
syri/main.nf:36:        2>| >(tee "${prefix}.error.log" >&2) \\
fastp/main.nf:50:            2>| >(tee ${prefix}.fastp.log >&2) \\
fastp/main.nf:71:            2>| >(tee ${prefix}.fastp.log >&2)
fastp/main.nf:96:            2>| >(tee ${prefix}.fastp.log >&2)
purgedups/calcuts/main.nf:29:    calcuts $args $stat > ${prefix}.cutoffs 2>| >(tee ${prefix}.calcuts.log >&2)
purgedups/purgedups/main.nf:30:        $paf > ${prefix}.dups.bed 2>| >(tee ${prefix}.purge_dups.log >&2)
bbmap/bbsplit/main.nf:91:        $args 2>| >(tee ${prefix}.log >&2)
kallisto/quant/main.nf:60:            ${reads} 2>| >(tee -a ${prefix}/kallisto_quant.log >&2)

I'm surprised it is used so often, since it actually removes any output from stderr, rather than redirects it to the pipe. So with it, one might miss important error output. One can test this by e.g. .trying to cat a file that does not exist, and pipe to less, to see if the error message is shown:

$ cat idontexist.txt | echo

cat: idontexist.txt: No such file or directory
$ cat idontexist.txt 2>| echo
$

(So, when we do

2>|

the error message is not forwarded). This might very well be intended, but I just got a little suspicious when it is used so often, and wondered if sometimes what was intended was

2>&1 |

, which will merge stderr and stdout. Any comments?

The original issue https://github.com/nf-core/modules/issues/8455

@Mahesh Binzer-Panchal Ah, now I see, and now I also realize that the commands after (

>(COMMAND etc)

) are in fact file descriptors rather than just a command that is piped to. All makes sense now, thank you!

@Marcel Ribeiro-Dantas had a nice explanation for this

Well that was a different question Maxime, it was why it was being used when the tool didn't seem to need it 😉

Similar enough

I think it would be better practice if the developer actually tested which stream a tool wrote to, and not mix by default. If a tool is writing to STDOUT, there's no need to catch STDERR which could start to include other things like we had with singularity at some point.