Hello, I have an issue regarding the step: ALIGN:ALIGN_BWA_BWAMEM2_BWAMEME. This step failed on many cases:

[93/255ffe] NFCORE_RAREDISEASE:RAREDISEASE:ALIGN:ALIGN_BWA_BWAMEM2_BWAMEME:BWAMEM2_MEM (SRR8082060_LNUMBER95) [  0%] 0 of 179, retries: 73

I go to the

work/

directory to check some of the error message. So far as I see the error message is always:

samtools sort: truncated file. Aborting

I checked raw fastq files and they are not broken. I also tried to run bwa + samtool in two steps outside of the pipeline using some of the failed samples. I don't see the error and the sorted bam files can be generated successfully. I'm wondering if any of you happen to know how to solve this issue? Many thanks in advance.

Do you still have this issue @Jiasheng If you are able to run the processes outside of the workflow, this could be a resources issue. Perhaps the allocated amount of memory if too low?

I've solved the issue. I tried to run the process outside of the workflow, but still didn't work. Then, I split the two steps - first

bwa-mem2

then

samtools

, and finally it turned out the problem happened at the first step. There's something wrong with

bwa-mem2

its internal AVX-512 build, which can cause memory corruption or segmentation faults. After switching from the AVX-512 build to the AVX2 build, the alignment runs stabilized and these memory-corruption issues disappeared. I actually cloned the

nf-core/raredisease

pipeline locally and also modified this step (from a streaming pipeline to a two-step process). Since then, I haven’t seen a single error. This entire debugging process took me almost two months, and I hope this workaround can help someone else facing the same issue.