https://nf-co.re logo
Join Slack
Powered by
I also noticed something when I was running the jo...
# methylseq
k
I also noticed something when I was running the job one by one - when aligning with bismark, I am seeing a lot of sample1_val_1.fq.gz.temp.0... uptil temp.3 for each R1 and R2 read. I am also seeing sample1_val_1.fq.gz.temp.0_G_to_A.fastq up till temp 3. The bismark runs with 4 cores, is that why I am seeing these files? But why are they fastq, and not fastq.gz? They are each 30gb in size 🥲👌
p
The 
.temp.N
files are for the multi-core processing. If you don't use parallel processing (only 1 core) then they won't be created.
The 
_C_to_T
and 
_G_to_A
files are unavoidable though
This might be something we could avoid in the Rust rewrite of Bismark though.. 🤔 Looking into it now.
Would be nice to avoid writing these huge intermediate files, I agree 👍🏻
Ok, I think it should be possible to stream everything and avoid these intermediate files entirely in the Rust rewrite of Bismark (not hisat2, and only on systems that can handle FIFO handles).
Done! Benchmarking complete: zero intermediate files, same speed and memory usage
Just need @Felix Krueger to get back from holiday and review + merge + release now!
🙌 2
Open in Slack
PreviousNext
Powered by