OK I think I may be tracking down (at least one source of) the problem with Aspera for me. It seems to be coming down, at least in part, to improperly specified MD5 checksums. Here's one file that's failed multiple times: If I peek at the

command.sh

file and run the Aspera command manually, it completes and gives me an md5 starting with 7b7:

(aspera) jonsan@nf-head:~/fetchngs/EA_pharma/fetchngs_exec/test$ ascp     -QT -l 300m -P33001     -i $CONDA_PREFIX/etc/aspera/aspera_bypass_dsa.pem     <mailto:era-fasp@fasp.sra.ebi.ac.uk|era-fasp@fasp.sra.ebi.ac.uk>:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz     SRX13191258_SRR17001000_1.fastq.gz
SRX13191258_SRR17001000_1.fastq.gz                                                                                                                 100%   26MB 15.3Mb/s    00:10
Completed: 26745K bytes transferred in 11 seconds
 (19634K bits/sec), in 1 file.
(aspera) jonsan@nf-head:~/fetchngs/EA_pharma/fetchngs_exec/test$ md5sum SRX13191258_SRR17001000_1.fastq.gz                                                                           7b7e0af5429bcb54b2c232489ea8212b  SRX13191258_SRR17001000_1.fastq.gz

However, the

command.sh

file appears to be checking it with a different checksum:

#!/bin/bash -euo pipefail
ascp \
    -QT -l 300m -P33001 \
    -i $CONDA_PREFIX/etc/aspera/aspera_bypass_dsa.pem \
    <mailto:era-fasp@fasp.sra.ebi.ac.uk|era-fasp@fasp.sra.ebi.ac.uk>:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz \
    SRX13191258_SRR17001000_1.fastq.gz

echo "3fcee2e72a2ec6221cac142538aff092  SRX13191258_SRR17001000_1.fastq.gz" > SRX13191258_SRR17001000_1.fastq.gz.md5
md5sum -c SRX13191258_SRR17001000_1.fastq.gz.md5

When I take a look at this run's metadata file, I actually see both of those md5 checksums present in different columns:

fastq_md5	7b7e0af5429bcb54b2c232489ea8212b;3fcee2e72a2ec6221cac142538aff092;383df08e03e1cd1ee071fd67c16b085b
fastq_bytes	27387589;1445187226;1481254395
fastq_ftp	<http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz|ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz>
fastq_galaxy	<http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz|ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz>
fastq_aspera	<http://fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz|fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz;fasp.sra.ebi.ac.uk:/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz>
fastq_1	<http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz|ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_1.fastq.gz>
fastq_2	<http://ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz|ftp.sra.ebi.ac.uk/vol1/fastq/SRR170/000/SRR17001000/SRR17001000_2.fastq.gz>
md5_1	3fcee2e72a2ec6221cac142538aff092
md5_2	383df08e03e1cd1ee071fd67c16b085b

It seems as if there might be some sort of bug being introduced by the presence of three different paths in the

fastq_X

columns?

nice investigation

It’s very odd, I’ve been attempting to download a number of studies and I’m getting quite a range of failure rates. Haven’t fully investigated each one yet but I’m surprised at how variable it seems to be.