Help: Limiting CPU Usage with Local Executor and A...
# configs
g
Help: Limiting CPU Usage with Local Executor and Apptainer Hi everyone, I'm currently running the nf-core RNA-Seq pipeline on a server using the local executor and Apptainer. Although I've set CPU limits in the Nextflow config file, tasks like STAR alignment are consuming all available CPU cores rather than following the specified limits in my config file. The Nextflow documentation mentions that "it is possible for a local task to use more CPUs and memory than it requested" with a local executor but it doesn't provide a solution. Could someone help me figure out how to restrict the Nextflow run to my specified parameters? My nextflow.config file:
Copy code
params { 
max_memory = 40.GB 
max_cpus = 8 
max_time = 24.h }

executor { 
name = 'local' 
queueSize = 5 }

resume = true

apptainer { 
enabled = true 
autoMounts = 
true pullTimeout = '4h' }

process { 
errorStrategy = 'retry' 
maxRetries = 3 
container = 'rnaseq.sif' }
p
You should adjust that outside of nextflow (e.g. using
taskset
if you just want to select cores, or by configuring the corresponding cgroup.
s
max_memory
and friends are the older format, have you tried using resourceLimits?
p
Those only matter for the dead counting anyway, if using the
local
executor and singularity or apptainer, there's nothing enforcing those limits (the typical expectation would be that you have a scheduler that sets cgroups limits if you can't use the whole node, otherwise the whole node should be fine).