Hello all, I'm new to nextflow (coming from a Snakemake background) so I'm trying to get a feel for how these pipelines work. I'm trying to run a small sample set through the smrnaseq pipeline locally but keep running into this error: (I have access to an HPC, but have not been able to get the test data provided by the pipeline to work with singularity and have a time crunch. The test profile works with docker locally for me, so thats how I am rolling for now...)

Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_SMRNASEQ:CONTAMINANT_FILTER:INDEX_TRNA (hg38-tRNAs.fa)'

Caused by:
  Process requirement exceeds available memory -- req: 72 GB; avail: 64 GB

I've tried a few things to get around this: 1. Included a custom config with -c: (same error)

# local.config
process {
  cpus   = 8
  memory = 62.GB
}

2. Added the following line to nextflow.config (same error)

process {
    memory = '62.GB'

The command I am using to run the pipeline is as follows:

nextflow run nf-core/smrnaseq -profile docker -params-file params.yaml

My params.yaml is as follows:

input: './samplesheet.csv'
outdir: './results'
genome: 'GRCh38'
mirtrace_species: 'hsa'
save_intermediates: true
filter_contamination: true
trna: '<http://gtrnadb.ucsc.edu/genomes/eukaryota/Hsapi38/hg38-tRNAs.fa>'
cdna: '<ftp://ftp.ensembl.org/pub/release-86/fasta/homo_sapiens/cdna/Homo_sapiens.GRCh38.cdna.all.fa.gz>'
ncrna: '<ftp://ftp.ensembl.org/pub/release-86/fasta/homo_sapiens/ncrna/Homo_sapiens.GRCh38.ncrna.fa.gz>'

If anyone has any insight as to how I can fix this that would be great!

How much RAM and CPUs do you have "locally"? Because the

test

profile does not run with real world data and just test some basics (like that Nextflow and Docker works, etc.). Depending on the answer, you may have a better shot at using the HPC. (What issues did you see with Singularity?) No matter where you run, though, you'll have better success if you tell Nextflow what it should be working with -- local executor or the HPC scheduler (SLURM, LSF, etc.). Docker or Singularity/Apptainer, etc., how much RAM and how many CPUs are available, as well as time limits (usually on the HPC). If you don't do that Nextflow will run in the most primitive way possible and make some (potentially wrong) assumptions, so investing some time to get the config right is a good idea. Here are some tips. If you want to use a custom config file (as alternative to or in addition to a profile), I recommend using it with the

-c

parameter, rather than relying on mechanisms such as a

nextflow.config

in a specific directory -- that you way there are no side effects and it is is obvious which config you are using. Regarding your config attempts: • Every time you use

process { ... }

without a qualifier such as

withName

or

withLabel

, it applies to all processes, which is a bad at least for resource allocation, because processes can have very different requirements. • You are trying to control the actual allocation with

process.cpus

and

process.memory

. This basically overrides what the pipeline wants to allocate. Sometimes that can be required, but I would start with giving the pipeline limits within which to operate. In newer Nextflow versions, this is done with a

resourceLimits

directive. • Note that on a local machine (without a scheduler that enforces limits) any allocation requests are not really binding -- the actual software can use as many CPUs and as much RAM as it can get a hold of. (The allocated CPUs are often passed to the sofware as a "threads" parameter, but for RAM the software allocates what it thinks it will need, which could be more than you have available, although Docker can enforce some limits.) Regarding your parameters: I don't know a lot about smRNAseq, although I have run different versions of this pipeline a couple of times successfully. I have used the parameters

fasta

, as well as

--mirgenedb

,

--mirgenedb_species

,

--mirgenedb_gff

,

--mirgenedb_mature

, and

--mirgenedb_hairpin

(the latter files were all from https://www.mirgenedb.org/). When you are mixing different sources, you have to be careful that all the contig names, etc., are compatible. (My guess is that the tRNA file from UCSC has

chr

prefixed contigs, whereas the Ensemble files don't; not sure if it matters in this case, but you should find out.)

Thanks Thomas! Really helpful advice! Thankfully we were able to get the pipeline to run locally on a more high-RAM machine, but I think now that we have some information, we will go back and try to correctly implement on an HPC. Much appreciated!!