https://nf-co.re logo
Join Slack
Powered by
Hi there! I am having a weird issue with mag runni...
# mag
r
Hi there! I am having a weird issue with mag running on slurm. I am interested in the bacterias in a a metagenome, so I run mag excluding the host genome, the script is
Copy code
#!/bin/bash
#SBATCH --job-name=nfcore_mag
#SBATCH --partition=main
#SBATCH --time=24:00:00
#SBATCH --cpus-per-task=16
#SBATCH --mem=128G
#SBATCH --output=/scratch1/liruiqi/meta_Osedax/mag_outputs/logs/nfcore_mag.%j.out
#SBATCH --error=/scratch1/liruiqi/meta_Osedax/mag_outputs/logs/nfcore_mag.%j.err

# activate conda environment
source ~/.bashrc
conda activate meta
# run nf-core/mag
nextflow run nf-core/mag \
  -profile conda \
  --input samplesheet.csv \
  --outdir mag_outputs \
  --spades_fix_cpus 16 \
  --host_fasta hostgenome.fna \
  --host_removal_verysensitive
however, mag did assemble the metagenome with SPAdes, but it stopped at the binning step, the direct cause seem to be a samtool version issue.
Copy code
SPAdes/
total 903K
-rw-rw-r-- 1 liruiqi 175K Feb  7 21:21 SPAdes-CR180008.spades.log
-rw-rw-r-- 1 liruiqi  31M Feb  7 21:21 SPAdes-CR180008.scaffolds.fa.gz
-rw-rw-r-- 1 liruiqi  31M Feb  7 21:21 SPAdes-CR180008.contigs.fa.gz
-rw-rw-r-- 1 liruiqi  46M Feb  7 21:21 SPAdes-CR180008.assembly.gfa.gz
slurm log:
Copy code
Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Error executing process > 'NFCORE_MAG:MAG:BINNING_PREPARATION:SHORTREAD_BINNING_PREPARATION:BOWTIE2_ASSEMBLY_ALIGN (SPAdes-CR180008-CR180008)'

Caused by:
  Process `NFCORE_MAG:MAG:BINNING_PREPARATION:SHORTREAD_BINNING_PREPARATION:BOWTIE2_ASSEMBLY_ALIGN (SPAdes-CR180008-CR180008)` terminated with an error exit status (1)


Command executed:

  INDEX=`find -L ./ -name "*.rev.1.bt2l" -o -name "*.rev.1.bt2" | sed 's/.rev.1.bt2l//' | sed 's/.rev.1.bt2//'`
  bowtie2 \
      -p "2" \
      -x $INDEX \
       \
      -1 "CR180008_run0_phix_removed.unmapped_1.fastq.gz" -2 "CR180008_run0_phix_removed.unmapped_2.fastq.gz" \
      2> "SPAdes-CR180008-CR180008.bowtie2.log" |         samtools view -@ "2" -bS |         samtools sort -@ "2" -o "SPAdes-CR180008-CR180008.bam"
  samtools index "SPAdes-CR180008-CR180008.bam"

  if [ SPAdes-CR180008-CR180008 = "SPAdes-CR180008-CR180008" ] ; then
      mv "SPAdes-CR180008-CR180008.bowtie2.log" "SPAdes-CR180008.bowtie2.log"
  fi

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MAG:MAG:BINNING_PREPARATION:SHORTREAD_BINNING_PREPARATION:BOWTIE2_ASSEMBLY_ALIGN":
      bowtie2: $(echo $(bowtie2 --version 2>&1) | sed 's/^.*bowtie2-align-s version //; s/ .*$//')
      samtools: $(echo $(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*$//')
      pigz: $( pigz --version 2>&1 | sed 's/pigz //g' )
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  view: invalid option -- '@'
  sort: invalid option -- '@'
  [main_samview] fail to open "2" for reading.
  open: No such file or directory
  [bam_sort_core] fail to open file 2

Work dir:
  /scratch1/liruiqi/meta_Osedax/mag_outputs/work/4e/b9ed6852bec91d92e8954832213e4b

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>
Would you mind helping me troubleshooting this problem?
Thank you very much!
btw the two raw reads files fastq.gz is ~1.5GB each
j
Sorry @Ruiqi Li I did see your github issue, but only was able to get to it now
I've not seen this error before, it look sa bit odd
Maybe like it did not pick up the indexed assembly FASTA file properlyl...
Do you still have the directopry
Copy code
/scratch1/liruiqi/meta_Osedax/mag_outputs/work/4e/b9ed6852bec91d92e8954832213e4b
?
r
Copy code
lss /scratch1/liruiqi/meta_Osedax/mag_outputs/work/4e/b9ed6852bec91d92e8954832213e4b
total 5.0K
lrwxrwxrwx 1 liruiqi 109 Feb  7 21:23 SPAdes-CR180008.scaffolds.fa -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/38/128dff41b70b236ae0e07a40d71fe7/SPAdes-CR180008.scaffolds.fa
-rw-rw-r-- 1 liruiqi  49 Feb  7 21:23 SPAdes-CR180008-CR180008.bowtie2.log
lrwxrwxrwx 1 liruiqi 127 Feb  7 21:23 CR180008_run0_phix_removed.unmapped_2.fastq.gz -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/fa/d889cbad7ca665421f4d4b5903ba80/CR180008_run0_phix_removed.unmapped_2.fastq.gz
lrwxrwxrwx 1 liruiqi 127 Feb  7 21:23 CR180008_run0_phix_removed.unmapped_1.fastq.gz -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/fa/d889cbad7ca665421f4d4b5903ba80/CR180008_run0_phix_removed.unmapped_1.fastq.gz
lrwxrwxrwx 1 liruiqi 105 Feb  7 21:23 bt2_index_base.rev.2.bt2 -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/5e/83eb4b99ce2151794d8a0afc4416a8/bt2_index_base.rev.2.bt2
lrwxrwxrwx 1 liruiqi 105 Feb  7 21:23 bt2_index_base.rev.1.bt2 -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/5e/83eb4b99ce2151794d8a0afc4416a8/bt2_index_base.rev.1.bt2
lrwxrwxrwx 1 liruiqi 101 Feb  7 21:23 bt2_index_base.4.bt2 -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/5e/83eb4b99ce2151794d8a0afc4416a8/bt2_index_base.4.bt2
lrwxrwxrwx 1 liruiqi 101 Feb  7 21:23 bt2_index_base.3.bt2 -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/5e/83eb4b99ce2151794d8a0afc4416a8/bt2_index_base.3.bt2
lrwxrwxrwx 1 liruiqi 101 Feb  7 21:23 bt2_index_base.2.bt2 -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/5e/83eb4b99ce2151794d8a0afc4416a8/bt2_index_base.2.bt2
lrwxrwxrwx 1 liruiqi 101 Feb  7 21:23 bt2_index_base.1.bt2 -> /scratch1/liruiqi/meta_Osedax/mag_outputs/work/5e/83eb4b99ce2151794d8a0afc4416a8/bt2_index_base.1.bt2
Also, I only saw the SPAdes assembly in my Assembly folder, I guess that's because the pipeline decided SPAdes assembly is better than MEGAHIT one?
j
Can you share teh bowtie2.log?
r
Copy code
cat SPAdes-CR180008-CR180008.bowtie2.log
(ERR): bowtie2-align died with signal 13 (PIPE)
j
Well that's useless 😅
Thank you bowtie...
Could you 
zcat <file> | head
all the FASTQ files and head the scaffolds file?
r
do you mean all FASTQ files in that dir?
Copy code
zcat CR180008_run0_phix_removed.unmapped_2.fastq.gz | head
@lh00134:648:22MTTJLT4:2:1101:18369:1126 2:N:0:GTACGCATGA+ACGCTATGTC
ACTGACTGACTGATTAATTAACTGACTGATTAACTAACTGACTGATTAACTGACTGACTGATTAATTAACTGACTGATTAACTAACTGACTGATTAACTAACTGATTAATTAACTGACTGATTAACTGACTGACTGATTAACTGACTGAC
+
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII9I*IIIIIIIIIIIIIIIII9I9IIIIIIIIIIIIII*IIIIIII
@lh00134:648:22MTTJLT4:2:1101:21363:1126 2:N:0:GTACGCATGA+ACGCTATGTC
AAATAAGAATATGGACAAATCAGTACAAATCAATTAATGGAATATTGTGAAATATTCCATTAGAAGTATATACAATATCTAATATGCATGAAAACGAGTCTGGATTAAATTATTCATGAGACTAAATAAATATGTGTAAGTAATTTAGAC
+
IIIIIIIIIIIIIIIII*I9IIIIIII9IIIIIIIIIIIIIIII9IIIIIIIIIIII99II9IIIIIIIIIIIII9IIIIIIIIIIIIIIIII9I9III99IIIIIII9I999IIIIII9IIIIIIII9*II9IIIII9II99III*III
@lh00134:648:22MTTJLT4:2:1101:4980:1252 2:N:0:GTACGCATGA+ACGCTATGTC
TAATTAACTGACTGATTAACTAACTAAGTGATTAACTGACTGACTGATTAACTGACTGACTGATTAATTAACTGACTGATTAACTGACTGACTGATTAACAGACTGACTGATTAATAAACTGACTGATTAACTGACTGACTGATTAACTG
[liruiqi@discovery1 b9ed6852bec91d92e8954832213e4b]$ zcat CR180008_run0_phix_removed.unmapped_1.fastq.gz | head
@lh00134:648:22MTTJLT4:2:1101:18369:1126 1:N:0:GTACGCATGA+ACGCTATGTC
GTTAGTTAATCAGTCAGTTAATTAATCAGTCAGTCGGTTAATCAGTCAGTTAGTTAATCAGTCAGTCAGTTAATCAGTCAGTCAGTTAATCAGTCAGTTAGTTAATCAGTCAGTTAATTAATCAGTCAGTCAGTTAATCAGTCAGTTAGT
+
IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII9IIIIIIIIIIIIIIIIIIIIIII
@lh00134:648:22MTTJLT4:2:1101:21363:1126 1:N:0:GTACGCATGA+ACGCTATGTC
CTTCCACCGGGTATCAAACTCAGCTCGATTTCTCATGTCATACTTCCTCTGTAGTTCGCGCTGCCAATTATTCCCTTTATCAAATGTCTAAATTACTTACACATATTTATTTAGTCTCATGAATAATTTAATCCAGACTCGTTTTCATGC
+
II9I9I9IIIIIIIII9IIIII99IIIIIIIIII9IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII*IIIII*I9*IIIIIIII9I9IIIIIIIIIIIIII9IIIIII99III9*I*IIIIIII*IIIIIIIIIII9II9IIII
@lh00134:648:22MTTJLT4:2:1101:4980:1252 1:N:0:GTACGCATGA+ACGCTATGTC
AGTTAATCAGTCAGTTAATTAATCAGTCAGTCAGTTAATCAGTCAGTTAGTTAATCAGTCAGTTAGTTAATCAGTCAGTCAGTTAATCAGTCAGTCAGTTAATCAGTCAGTTTATTAATCAGTCAGTCTGTTAATCAGTCAGTCAGTTAA
[liruiqi@discovery1 b9ed6852bec91d92e8954832213e4b]$
Copy code
head SPAdes-CR180008.scaffolds.fa
>NODE_1_length_110675_cov_93.367863
TAAACGCTTTGTTGGATAGTCGCCAAGTTGAGCGGAGTATGATGGTCAATCAGGTTAATA
AATACCACACTCCAGCCCGCAGGCTAGATGTTCGCCTAAGCTCTCACAGTCAGCCATAAA
GCTAACCTGCCACTCTCCTCTGCAAATCAGTTCCGGCTGCACTTGCTTCCACTTCAAGCC
TTTACAGATGCTCTGTATCGCTTTTTGGGTGCCGGTTCCATCGTGACCGGCGCGGATGTA
AAAGCCGAACGGCATGCCTTGCTTGGCTTCTAAACAAGGGTAGTAGATGCGGTCGAAAAA
GTCTTTAAGGGCGCCGCTTATATAGCCTAAATTTTCTGTCGTACCGAGAATGATAGCGTC
ACAGGACAGTACATCAGCAGCGGTGGCTTGTAGTGGTGGGATCCATTCAATCACTAAATT
GATTTCACCTTTGCGGCAGTTTGCATTGACTCTTTGCGCGCCTGTCAGGGTAGCTGTAGC
CAGTTGCTGAGTATTGCTTGATGGAGCGTGGGCCACTATTAATAGCTTTTTCATTGCGGT
j
Ok so they all have contents?
r
yeah
2 fastq files and 1 fasta all have contents
j
Ok next debugging step: could you try running the 
find
command at the beginning of the script in the error message?
r
do you think it is a input file issue, or a HPC issue? it can't be a software issue right since all softwares are from mag nf pipeline
j
It may be, but the error message is really weird
I can't remember now - you're running this with conda or singularity/apptainer?
Ah conda
r
[liruiqi@discovery1 b9ed6852bec91d92e8954832213e4b]$ find -L ./ -name "*.rev.1.bt2l" -o -name "*.rev.1.bt2" ./bt2_index_base.rev.1.bt2
yeah conda
Copy code
find -L ./ -name "*.rev.1.bt2l" -o -name "*.rev.1.bt2" | sed 's/.rev.1.bt2l//' | sed 's/.rev.1.bt2//'
./bt2_index_base
j
If you do 
samtools --version
what happens?
But yeah... If you've not already maybe you can try from scratch and see what hapoend?
r
Copy code
conda activate meta
(/scratch1/liruiqi/condaEnvs/meta) [liruiqi@discovery1 ~]$ samtools --version
samtools 1.23
Using htslib 1.23
Copyright (C) 2025 Genome Research Ltd.

Samtools compilation details:
    Features:       build=configure curses=yes
    CC:             /opt/conda/conda-bld/samtools_1765917324003/_build_env/bin/x86_64-conda-linux-gnu-cc
    CPPFLAGS:       -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /scratch1/liruiqi/condaEnvs/meta/include
    CFLAGS:         -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /scratch1/liruiqi/condaEnvs/meta/include -fdebug-prefix-map=/opt/conda/conda-bld/samtools_1765917324003/work=/usr/local/src/conda/samtools-1.23 -fdebug-prefix-map=/scratch1/liruiqi/condaEnvs/meta=/usr/local/src/conda-prefix
    LDFLAGS:        -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/scratch1/liruiqi/condaEnvs/meta/lib -Wl,-rpath-link,/scratch1/liruiqi/condaEnvs/meta/lib -L/scratch1/liruiqi/condaEnvs/meta/lib
    HTSDIR:
    LIBS:
    CURSES_LIB:     -ltinfow -lncursesw

HTSlib compilation details:
    Features:       build=configure libcurl=yes S3=yes GCS=yes libdeflate=yes lzma=yes bzip2=yes plugins=yes plugin-path=/scratch1/liruiqi/condaEnvs/meta/libexec/htslib htscodecs=1.6.5
    CC:             /opt/conda/conda-bld/htslib_1765902480168/_build_env/bin/x86_64-conda-linux-gnu-cc
    CPPFLAGS:       -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /scratch1/liruiqi/condaEnvs/meta/include
    CFLAGS:         -Wall -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /scratch1/liruiqi/condaEnvs/meta/include -fdebug-prefix-map=/opt/conda/conda-bld/htslib_1765902480168/work=/usr/local/src/conda/htslib-1.23 -fdebug-prefix-map=/scratch1/liruiqi/condaEnvs/meta=/usr/local/src/conda-prefix -fvisibility=hidden
    LDFLAGS:        -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/scratch1/liruiqi/condaEnvs/meta/lib -Wl,-rpath-link,/scratch1/liruiqi/condaEnvs/meta/lib -L/scratch1/liruiqi/condaEnvs/meta/lib -fvisibility=hidden -rdynamic

HTSlib URL scheme handlers present:
    built-in:	 file, preload, data
    Amazon S3:	 s3+https, s3, s3+http
    libcurl:	 gophers, smtp, wss, smb, rtsp, tftp, pop3, smbs, imaps, pop3s, ws, ftps, https, ftp, gopher, sftp, imap, http, smtps, scp, dict, mqtt, telnet
    Google Cloud Storage:	 gs+http, gs+https, gs
    crypt4gh-needed:	 crypt4gh
    mem:	 mem
the same issue happened again
theoretically, you would encounter the same issue if you run the same command with the same files on your end with conda, right? I am also happy to share the files
they are only 2 1.5GB fastq.gz and a 300mb genome
Hi James, anything else you need for troubleshooting?
do you think this could be related to our HPC?
beegfs_nodev on /scratch1 type beegfs (rw,nosuid,relatime,cfgFile=/etc/beegfs/beegfs-client-scratch1.conf,_netdev)
j
Sorry, drowning in deadlines atm (and late in my day) I would be surprised if it's the file system.... My only idea currently is to break down the command try manually run the bowtie2 command first and see if that works
(outside the pipeline)
r
it actually might be a file system issue: I run it on another allocation on our HPC and that error didn't show up!
thank you!
by the way, I am running it for a two 1.5GB fastq.gz short reads file, how much cpu, mem and wall time would you recommend?
j
Oh ok! Actually maybe that does make sense... If therr is a hiccup with the FS, maybe bowtie can't pick up the files correctly and then pipes nonsense or something? I 'hope ' it is that as I'm running out of ideas other than thr manual execution debugging I mentioned above 😅 Sorry for the slow replies again! For the wall time, I normally base on number of reads rather than file size. But I think the defaults should be ok.... But it really depends on the complexity and host content etc so it's sort of hard to predict
2 Views
Open in Slack
PreviousNext
Powered by