Hi everyone! I'm trying to download FASTQ files from the recent project PRJEB66187 (ENA).

nextflow run nf-core/fetchngs -r 1.12.0 --input kalin2024_PRJEB66187_sra.tsv -profile docker --nf_core_pipeline rnaseq --outdir kalin2024_PRJEB66187/

However I get an error with non-matching checksums (for all samples that I have tried thus far):

ERROR ~ Error executing process > 'NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP (ERX11497477_ERR12120410)'

Caused by:
  Process `NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP (ERX11497477_ERR12120410)` terminated with an error exit status (1)

Command executed:

  wget \
      -t 5 -nv -c -T 60 \
      -O ERX11497477_ERR12120410_1.fastq.gz \
      ftp.sra.ebi.ac.uk/vol1/fastq/ERR121/010/ERR12120410/ERR12120410_1.fastq.gz
  
  echo "9d23780a1911de295da2b8d57f701f15  ERX11497477_ERR12120410_1.fastq.gz" > ERX11497477_ERR12120410_1.fastq.gz.md5
  md5sum -c ERX11497477_ERR12120410_1.fastq.gz.md5
  
  wget \
      -t 5 -nv -c -T 60 \
      -O ERX11497477_ERR12120410_2.fastq.gz \
      ftp.sra.ebi.ac.uk/vol1/fastq/ERR121/010/ERR12120410/ERR12120410_2.fastq.gz
  
  echo "79c49f48752e1db1dc4d78a970ec49a2  ERX11497477_ERR12120410_2.fastq.gz" > ERX11497477_ERR12120410_2.fastq.gz.md5
  md5sum -c ERX11497477_ERR12120410_2.fastq.gz.md5
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_FETCHNGS:SRA:SRA_FASTQ_FTP":
      wget: $(echo $(wget --version | head -n 1 | sed 's/^GNU Wget //; s/ .*$//'))
  END_VERSIONS

Command exit status:
  1

Command output:
  ERX11497477_ERR12120410_1.fastq.gz: FAILED

Command error:
  2024-03-01 11:16:43 URL:<http://ftp.sra.ebi.ac.uk/vol1/fastq/ERR121/010/ERR12120410/ERR12120410_1.fastq.gz> [3036937771/3036937771] -> "ERX11497477_ERR12120410_1.fastq.gz" [1]
  md5sum: WARNING: 1 of 1 computed checksums did NOT match
  ERX11497477_ERR12120410_1.fastq.gz: FAILED

I have manually downloaded a few samples from this project and the md5 matched the one mentioned in the "fastq_md5" column from the ENA website (https://www.ebi.ac.uk/ena/browser/view/PRJEB66187). I don't know where the md5 obtained from the pipeline come from? Edit: the .md5 file from the pipeline is correct and matches the one from the ENA website and the one I get when downloading the FASTQ manually. It is just that the FASTQ from the pipeline doesn't match the associated md5. Thanks!

Hmmm...this is weird.

The test profile runs perfectly. I tried to pull data for samples from another ENA project right now and it works as well. Maybe the issue is somehow with the PRJEB66187 project itself

It could be 🤔

I did check manually!

2024-03-01 10:15:20 URL:<http://ftp.sra.ebi.ac.uk/vol1/fastq/ERR121/015/ERR12120815/ERR12120815_1.fastq.gz> [3865807360/3865807360] -> "ERX11497882_ERR12120815_1.fastq.gz" [4]
  md5sum: WARNING: 1 of 1 computed checksums did NOT match
  ERX11497882_ERR12120815_1.fastq.gz: FAILED

Pipeline:

cat *.md5
e5846c15c8dc27a4289fd99a15d819c0  ERX11497882_ERR12120815_1.fastq.gz

md5sum ERX11497882_ERR12120815_1.fastq.gz
f072f64268f82437c6d913772191ba3e  ERX11497882_ERR12120815_1.fastq.gz

Manual download:

md5sum ERR12120815_1.fastq.gz 
e5846c15c8dc27a4289fd99a15d819c0  ERR12120815_1.fastq.gz

But I ran this sample again and it works this time... It is very weird because I had this issue several times for different samples. Maybe it was just the checksums doing its work correctly and the downloaded files were indeed corrupted! My bad then 🫠

Yep, looks like a corrupted download indeed 👍🏽

It's working when only querying for few samples, as soon as there are many it fails (tried several times). But it doesn't seem to be related to a flaw in the pipeline!