I apologize if this is a trivial issue, but I cannot seem to get

nf-core/rnaseq

to use more of the available CPUs. I am trying to align 360 samples on a

c5.24xlarge

EC2 instance (96 CPUs). I tried to increase the nextflow queue size to 90, as well as set max cpus to 90, but when I check AWS CloudWatch, the CPU utilization seems to hover around 11% with an occasional spike to ~25%. My launch command is:

nextflow run nf-core/rnaseq \
    --input $DIR/samplesheet.csv \
    --outdir $DEST \
    --genome GRCh38 \
    --aligner star_rsem \
    --with_umi false \
    --skip_gtf_filter \
    --skip_biotype_qc \
    --skip_qualimap \
    --skip_deseq2_qc \
    -c $CONF \
    -resume \
    -qs 90 \
    -profile docker

where

$CONF

is

process {
    resourceLimits = [
        cpus: 90
    ]
}

Am I missing anything obvious? I saw that there used to be a

--max_cpus

flag, but it appears to no longer exist.

With resourceLimits, you are only setting the absolute maximum that a single process may take if required. The amount of CPUs a single process can take will vary depending on its process label. The associated configuration can be found here. STAR alignment has the

label 'process_high'

, which means aligning one sample will at max request 12 CPUS and 72 GiB of memory. I have no AWS experience, but if my quick Google search was correct, a c5.24xlarge machine should come with 96 vCPUs and 192 GiB of memory. That means that you can run 2-3 alignments at the same time only, because you exhaust your memory faster than your CPUs. Unfortunately, nf-core turned down my proposal to decouple memory from CPUs in configs, so you will have to provide a custom config yourself. Mind the config selector priority, so you need to redefine the

with_label:process_high

or directly

withName: STAR_ALIGN

.

Thanks, @Matthias Zepper. I played around with the

cpus

specifications as you suggested. However, I think the main issue is just the number of processes that Nextflow launches. The pipeline hasn’t even gotten to the STAR align step. It’s just been plowing through FASTQC for hours. The repo shows that FASTQC is process_medium, but whether I set

cpus = 1

or

cpus = 16

for that label, the pipeline does not seem to want to launch more than 16 processes at a time, as measured by

ps -ef | grep .command.run | wc -l

It appears that FASTQC itself doesn’t really utilize multiple CPUs, but then I would expect that setting

cpus = 1

would tell nextflow that it can launch 96 processes instead of the current 16 = 96 / 6 (where

cpus = 6

is the default you linked). I’ll keep playing with it. It’s possible that my setting is somehow not being used properly.

You can use

nextflow config

on your CLI command

nextflow config nf-core/rnaseq \ --input $DIR/samplesheet.csv ...

to display the final resolved configuration. Could it be that Docker limits the number of running containers to 16 by default?

Docker shouldn’t be an issue. It was running 48 processes during

FQ_LINT

, which again checks out, because that’s

process_low

with the default

cpus = 2

= 96/48.

nextflow -c $CONF config nf-core/rnaseq

shows that the custom values are being correctly incorporated, and

nf-core

itself lists the custom config file in its

configFiles

preamble. Yet, somehow the custom values don’t seem to have the effect I expect.

I'm not sure if I'm following but as Matthias implied earlier, you need to consider memory as well as cores.

process_medium

for rnaseq seems to default to 36 Gbytes so you wouldn't get very many simultaneously on a 192 Gbyte node.

Is there a reason you're not using AWS Batch? Running on single VM is fine for testing but if you're doing it with production workloads you're going to encounter issues like this repeatedly due to lack of a scheduler.

Purely opportunistic. Because of data downloading from the CRO, all sequences ended up on a single machine, so I thought I could bump the type up to

c5.24xlarge

and do the alignment “locally”. I’m switching it over to AWS Batch now…