I am trying to use nf-core/rnaseq to run a smartseq2 dataset and have several problems that I've come across that I'd appreciate help with firguring out. the first and most glaring one is the costs and time. Running the pipeline to process 1899 samples has taken over 4 days and a cost of $105 up to the ~1300th 'sample', which in my eyes is way too much. Having said that, I've never processed such data outside of an nf-core pipeline so I have no frame of reference. Some questions that spring to mind are: • Is the price and time taken normal? • How do I reduce time and price? • I Have been running the pipeline via nf-tower and using aws spots and the default spot called is .... Is this something that is inherent in the source code, or specified externally (which would me I should liaise with our engineering team). The second is the error encountered at around the 1300th sample where one sample couldnt be processed: Successful run: Name

NFCORE_RNASEQ:RNASEQ:QUANTIFY_STAR_SALMON:SALMON_QUANT (SRR6576774)

Command

salmon quant \
    --geneMap Mus_musculus.GRCm39.112.gtf \
    --threads 6 \
    --libType=IU \
    -t genome.transcripts.fa \
    -a SRR6576774.Aligned.toTranscriptome.out.bam \
     \
    -o SRR6576774

if [ -f SRR6576774/aux_info/meta_info.json ]; then
    cp SRR6576774/aux_info/meta_info.json "SRR6576774_meta_info.json"
fi

cat <<-END_VERSIONS > versions.yml
"NFCORE_RNASEQ:RNASEQ:QUANTIFY_STAR_SALMON:SALMON_QUANT":
    salmon: $(echo $(salmon --version) | sed -e "s/salmon //g")
END_VERSIONS

Failed run: Name

NFCORE_RNASEQ:RNASEQ:QUANTIFY_STAR_SALMON:SALMON_QUANT (SRR6576775)

Command

salmon quant \
    --geneMap Mus_musculus.GRCm39.112.gtf \
    --threads 16 \
    --libType=IU \
    -t genome.transcripts.fa \
    -a SRR6576775.Aligned.toTranscriptome.out.bam \
     \
    -o SRR6576775

if [ -f SRR6576775/aux_info/meta_info.json ]; then
    cp SRR6576775/aux_info/meta_info.json "SRR6576775_meta_info.json"
fi

cat <<-END_VERSIONS > versions.yml
"NFCORE_RNASEQ:RNASEQ:QUANTIFY_STAR_SALMON:SALMON_QUANT":
    salmon: $(echo $(salmon --version) | sed -e "s/salmon //g")
END_VERSIONS

Command error:

fatal error: An error occurred (404) when calling the HeadObject operation: Key "rnaseq/41/f40bf25396831ceec75b21ada67af7/genome.transcripts.fa" does not exist
  Version Info: This is the most recent version of salmon.
  # salmon (alignment-based) v1.10.1
  # [ program ] => salmon 
  # [ command ] => quant 
  # [ geneMap ] => { Mus_musculus.GRCm39.112.gtf }
  # [ threads ] => { 16 }
  # [ libType ] => { IU }
  # [ targets ] => { genome.transcripts.fa }
  # [ alignments ] => { SRR6576775.Aligned.toTranscriptome.out.bam }
  # [ output ] => { SRR6576775 }
  Logs will be written to SRR6576775/logs
  Library format { type:paired end, relative orientation:inward, strandedness:unstranded }
  [2024-08-13 16:35:19.628] [jointLog] [info] setting maxHashResizeThreads to 16
  [2024-08-13 16:35:19.628] [jointLog] [info] Fragment incompatibility prior below threshold.  Incompatible fragments will be ignored.
  [2024-08-13 16:35:19.628] [jointLog] [info] numQuantThreads = 10
  ============
  Exception : [The provided transcript file: "genome.transcripts.fa" does not exist!
  ]
  ============
  salmon alignment-quant was invoked improperly.
  For usage information, try salmon quant --help-alignments
  Exiting.

Lastly, assuming I resolve this issue, what is the best way of resuming the pipeline... Would it be possible to resume or do I have to start a new run with the remaining 599 samples or so and go from there... or, heaven forbid, is this run unredeemable 🥲???

• I run on an HPC with free compute, so I have no idea what things cost (lucky me!), but if I remember this correctly, in the rewriting the sarek pipeline, they reduced the cost from ~$75 per sample to ~$25. RNAseq is much less complex and cheaper, of course, but if I understand you correctly, you are paying less then 10 cents per sample, which seems not unreasonable. • Reducing cost & time: ◦ You can try to optimize the RAM & CPU allocations so that you are not requesting more powerful VMs than you need (since they cost more). ◦ Skip any optional steps that you are not using. ◦ Some methods/aligners may be more "effcient" than others. Certainly a pseudoaligner is much faster and cheaper than a "proper" alignment to the whole genome, but you will be forgoing a lot of valuable QC information that way (e.g. you won't know if a nonmapping read is intronic, intergenic, or some garbage/contamination). • For Seqera Platform/Tower, #C02TPTREJ03 is probably a good place to ask. There is also an #infrastructure-aws channel over on the Nextflow Slack. • Regarding the error you see, it seems that it can't find some files. Especially since you have processed other samples successfully, my best guess is that this is a temporary glitch in accessing files (maybe because of some time-out due to high load). I would retry with

-resume

and see what happens. 🤞 • Breaking your data set up into junks that you run separately is not necessarily bad, I don't know if you can usefully process a QC report with >1000 samples. The issue is that you may want all of the results in one table. A work-around would be to recreate the tables from the individual

quant.sf

files (you can use the script from the pipeline manually), or use those files with

tximport

directly to generate an R object for use with your favorite statistical package.

Like Thomas, I live in the blissful state of having "free" HPC access (although the CPU hours are billed to our department by the university's compute center), so I have little experience with running the pipeline on public cloud infrastructure. All I can tell is that your numbers are certainly not too much off from what a default run looks like. If you check the blog posts by Seqera und Microsoft Azure, which both benchmarked their setups based on this pipeline, then you see that they more or less end up with prices in the ballpark that you are operating in. With clever configuration, you can certainly bring down the costs somewhat, if you optimize your data handling quite significantly. But some of those savings, you then have to invest in obtaining a license. Other than that, you can certainly optimize the pipeline itself if you know that you will be processing so many samples. For the sake of modularity, the pipeline is split up in many processes, for which the input and output files need to be staged and are potentially compressed and decompressed over and over again. All of that can be avoided if you basically write a lean pipeline that has very little parameters and only runs exactly the tools you want and need.