With current release of pipeline, is it possible to globally cap CPU resource allocation? Running on a single workstation, not through SLURM. I had this before and it does not appear to work well. Using docker to handle all modules

process {
 resourceLimits = [
  cpus: 8
 ]
}

That is the way to do it for one process. If have e.g. 12 CPU and want the pipeline to only use 8, there is no way to do that. Nextflow can only request, the OS (normally a scheduler like SLURM) has to enforce it. Also, the pipeline is designed to farm jobs out to nodes, or VMs, or whatever, so it sets requirements, and a scheduler deals with the rest. That said, there is a parameter that might help you limit resource use on your machine.

process.maxForks

limits the number of instances of a particular process type at any given time. You can set that for a process by name, all processes, or by label, e.g.

withLabel: process_high

. The problem is that another type of process can still run in parallel and use up the remaining CPUs. A drastic solution could be always requesting more that half of the available CPUs (not sure how Nextflow determines what that is if you have hyperthreading enabled). If Docker does the enforcement properly (not sure how the OS itself handles that), it should be impossible to run more than one process at a time (since it wouldn't get the resources requested), and the CPU use would be limited to what that one process requested and to what Nexrflow itself needs (not that much). The other solution would be to install SLURM and configure it (and the pipeline for it), but that seems like a lot of work for what you are trying to achieve.

The executor will keep track and not submit jobs that don't make sense. If you're using a scheduler, you can probably rely on that to govern resources so it makes sense to submit all work you know about directly. When running locally, most tend to use the

local

executor, which will use dead counting to know if it makes sense to start more tasks. The

executor

scope contains multiple tunables to deal with that (https://www.nextflow.io/docs/latest/reference/config.html#executor), e.g.

cpus

and

memory

(note that these do not care about real values but rather the values from the nextflow process).

I have set cpus to 1 via nextflow resourcelimit. It would still process each read sets in parallel, however now per sample it is now only using 1 thread, which no longer sends the cpu to 100% usage

Whatever the avtal running respects that depends on the executor (where local does not) as well as possible container engine (where my memory says docker does).

For the CPUs the Nexflow request often has more influence, even in the absence of external enforcement, because whenever possible an option is set inside the process that directs the tool to use a number of threads equal to the the specified CPUs.