Hello! We have been having a weird (and transient)...
# rnaseq
k
Hello! We have been having a weird (and transient) issue with the HISAT2_ALIGN process that we have not been able to resolve. The main manifestation is that many (but not all) HISAT2 align processes fail with the following error:
Copy code
(ERR): mkfifo(/tmp/44.inpipe1) failed.
Exiting now ...
[main_samview] fail to read the header from "-".
We initially noticed this issue in the https://github.com/eQTL-Catalogue/rnaseq/ workflow (which is based on an earlier version of the nf-core/rnaseq workflow), but I have now also confirmed it with the latest version of the nf-core/rnaseq workflow. Notably, a sample that fails once might work just fine when the process is restarted. The only workaround that the we found so far is to downgrade Nextflow to version 22.04.3 (which completely eliminates the error), but using such an old Nextflow version is starting to become a problem now (e.g. nf-core/rnaseq does not support it). We are seeing this error in two independently maintained SLURM clusters (University of Tartu and University of Eastern Finland). Our issue might be related to these two: https://github.com/nf-core/rnaseq/issues/1270 https://github.com/nf-core/rnaseq/issues/1288 Since the HISAT2 output is piped into samtools, I tried removing this line from the process: https://github.com/nf-core/rnaseq/blob/master/modules/nf-core/hisat2/align/main.nf#L52 However, the error remained basically the same (without the [main_samview] part):
Copy code
(ERR): mkfifo(/tmp/45.inpipe1) failed.
Exiting now ...
This indicates that the pipe error occurs even when the only visible pipe is removed from process.
Ok, looks like I found the explanation and a solution: https://github.com/nextflow-io/nextflow/issues/5310
I also added the following comment to this GitHub issue: https://github.com/nf-core/rnaseq/issues/1270
I had the exact same issue with HISAT2 and was able to work around it by setting the NXF_SINGULARITY_NEW_PID_NAMESPACE=false environment variable.
The issue seems to stem from how the unique combination of Nextflow, singularity and HISAT2 handle the names of temporary files. See the excellent explanation from Martin Pollard under this Nextflow issue:
nextflow-io/nextflow#5310
An alternative would be to upgrade to HISAT v2.2.2 and manually specify the temp directory with --temp-directory: DaehwanKimLab/hisat2#466
j
Hi Kaur, long time no see! Thanks for the report and the solution!
👍 1