https://nf-co.re logo
Join Slack
Powered by
I am trying to run 2.6.0 version of methylseq on o...
# methylseq
p
I am trying to run 2.6.0 version of methylseq on our HPC. But I'm keep getting an error:
Copy code
Failed to close filehandle No space left on device
The alignment step seems to be writing temp files to the local node storage instead of the TMPDIR location and filling it up. Is there a way to redirect the tmp output of the alignment? I'm using Bismark as the aligner. Bismark seems to have a 
--temp_dir
but its not used in the nf-core pipeline.
s
hi you can set the 
--temp_dir
and also 
--gzip
options via 
ext.args
for the align module in 
conf/modules.config
and use it to save gzipped tmp file to a path you like
p
I added the temp_dir into the modules.config file like this
But I'm still getting the same error:
Copy code
Running Bismark Parallel version. Number of parallel instances to be spawned: 4

Current working directory is: /work/4578865.1.eeskin_pod_36.q/nxf.iLMc9V4b52

...

Created G -> A converted version of the FastQ file H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz.temp.1 (28442995 sequences in total)

Failed to close filehandle No space left on device

Created G -> A converted version of the FastQ file H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz.temp.2 (28442994 sequences in total)

Failed to close filehandle No space left on device

Created G -> A converted version of the FastQ file H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz.temp.3 (28442994 sequences in total)

Failed to close filehandle No space left on device

Created G -> A converted version of the FastQ file H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz.temp.4 (28442994 sequences in total)

Failed to close filehandle No space left on device
e
1. Could you post the config as a markdown block? 2. Could you explain what part of it is telling you that it’s fill up the local node storage?
p
Copy code
withName: BISMARK_ALIGN {
        ext.args = { [
            params.aligner == 'bismark_hisat' ? ' --hisat2' : ' --bowtie2',
            params.aligner == 'bismark_hisat' && params.known_splices ? " --known-splicesite-infile <(hisat2_extract_splice_sites.py ${known_splices})" : '',
            params.pbat ? ' --pbat' : '',
            ( params.single_cell || params.non_directional || params.zymo ) ? ' --non_directional' : '',
            params.unmapped ? ' --unmapped' : '',
            params.relax_mismatches ? " --score_min L,0,-${params.num_mismatches}" : '',
            params.local_alignment ? " --local" : '',
            '--temp_dir /u/scratch/a/ajcollie',
            !meta.single_end && params.minins ? " --minins ${params.minins}" : '',
            meta.single_end ? '' : (
                params.maxins ? " --maxins ${params.maxins}" : (
                    params.em_seq ? " --maxins 1000" : ''
                )
            )
        ].join(' ').trim() }
        publishDir = [
            [
                path: { "${params.outdir}/${params.aligner}/alignments/unmapped" },
                mode: params.publish_dir_mode,
                pattern: "*.fq.gz",
                enabled: params.unmapped
            ],
            [
                path: { "${params.outdir}/${params.aligner}/alignments" },
                mode: params.publish_dir_mode,
                pattern: "*.bam",
                enabled: (params.save_align_intermeds || params.skip_deduplication || params.rrbs)
            ],
            [
                path: { "${params.outdir}/${params.aligner}/alignments/logs" },
                mode: params.publish_dir_mode,
                pattern: "*.txt"
            ]
        ]
    }
Do you want the entire config?
This part of the log: 
Current working directory is: /work/4578865.1.eeskin_pod_36.q/nxf.iLMc9V4b52
is the local node storage. Right now I am playing the lottery hoping to get a node with enough local storage, 9/10 jobs have finished so far...
👍🏻 1
e
Gotcha, just didn't want to assume!
Have you tried setting
Copy code
env {
    TMPDIR = "/u/scratch/a/ajcollie"
}
Or is 
TMPDIR
set? Just wondering I had a recent issue with MultiQC suddenly not being able to use the tmpdir a few weeks ago...
Ah, I wonder if the nodes have a different tmpdir 🤔
p
Yes 
TMPDIR
is set. I was wondering if they had a different one. I tried 
TMP
, and 
TEMP
. Based on the docs https://www.hoffman2.idre.ucla.edu/Using-H2/Storage/Storage.html it is 
TMPDIR
And other rules output files into the set 
TMPDIR
e
Maybe try:
Copy code
process {
withName: BISMARK_ALIGN {
  scratch '$SCRATCH'
}
}
p
Still same error
😞 1
e
Ooo I wonder if BISMARK is picking up on the tmpdir
@Felix Krueger any other ideas?
s
@Peter Chovanec could you share the the 
.command.sh
from that workDir...I wanna see if its being set correctly at the end
e
Maybe we should turn this into an issue 😅
p
Copy code
#!/bin/bash -euo pipefail
bismark \
    -1 H9_naive_5iLA_hTSC_DNAme_rep1_1_val_1.fq.gz -2 H9_naive_5iLA_hTSC_DNAme_rep1_2_val_2.fq.gz \
    --genome BismarkIndex \
    --bam \
    --bowtie2 --multicore 4

cat <<-END_VERSIONS > versions.yml
"NFCORE_METHYLSEQ:METHYLSEQ:BISMARK:BISMARK_ALIGN":
    bismark: $(echo $(bismark -v 2>&1) | sed 's/^.*Bismark Version: v//; s/Copyright.*$//')
END_VERSIONS
Doesn't look like temp_dir was set
Also seems like the bedgraph conversion can also run out of space
Copy code
Now writing methylation information for file >>CpG_OT_H9_naive_5iLA_hTSC_DNAme_rep1_1_val_1_bismark_bt2_pe.deduplicated.txt.gz<< to individual files for each chromosome
  Failed to open filehandle: No space left on device at /usr/local/bin/bismark2bedGraph line 279, <IN> line 24495416.
  Warning: unable to close filehandle $temp_fhs{...} properly: No space left on device at /usr/local/bin/bismark2bedGraph line 279, <IN> line 24495416.
  Warning: unable to close filehandle $temp_fhs{...} properly: No space left on device at /usr/local/bin/bismark2bedGraph line 279, <IN> line 24495416.
  Warning: unable to close filehandle $temp_fhs{...} properly: No space left on device at /usr/local/bin/bismark2bedGraph line 279, <IN> line 24495416.
  Warning: unable to close filehandle $temp_fhs{...} properly: No space left on device at /usr/local/bin/bismark2bedGraph line 279, <IN> line 24495416.
  Warning: unable to close filehandle $temp_fhs{...} properly: No space left on device at /usr/local/bin/bismark2bedGraph line 279, <IN> line 24495416.
  Warning: unable to close filehandle $temp_fhs{...} properly: No space left on device at /usr/local/bin/bismark2bedGraph line 279, <IN> line 24495416.
  Warning: unable to close filehandle $temp_fhs{...} properly: No space left on device at /usr/local/bin/bismark2bedGraph line 279, <IN> line 24495416.
  Warning: unable to close filehandle $temp_fhs{...} pr
e
Hmm that's all pretty weird. Could you open an issue? Hoping to get all the info tracked. https://github.com/nf-core/methylseq/issues/new?assignees=&amp;labels=bug&amp;projects=&amp;template=bug_report.yml
f
Peter, what have you done?! I hope all is well (apart from running out of space?) :)
p
Hi Felix 😄 all is good, just helping out Mandy with her data.
I have opened an issue (let me know if I should add more detail) https://github.com/nf-core/methylseq/issues/413
Open in Slack
PreviousNext
Powered by