Hi, I am a bit puzzled about how Nextflow manages process resource allocation, and I would like to have your opinion about it (before creating an issue on Nextflow's Github if there really is a bug). Let's say we have this:

.
├── test.nf
├── test.config
└── bin/
    └── test.sh

with: • test.nf

#!/usr/bin/env nextflow

process shouldThrowOOM {

    cpus 4
    memory '1 GB'

    conda "conda-forge::python=3.14.2"
    container "${ workflow.containerEngine in ['singularity', 'apptainer'] && !task.ext.singularity_pull_docker_container ?
        '<https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/24/24d21af99eae8df9f2347bc7b137c8f10ccc905211c8a4e08761af491f3dd884/data>':
        'community.wave.seqera.io/library/python:3.14.2--41251566046f8583' }"

    input:


    script:
    """
    test.sh
    """
}

workflow {
    shouldThrowOOM( )
}

• test.config

profiles {
    conda {
        conda.enabled           = true
        docker.enabled          = false
        singularity.enabled     = false
        podman.enabled          = false
        shifter.enabled         = false
        charliecloud.enabled    = false
        conda.channels          = ['conda-forge', 'bioconda']
        apptainer.enabled       = false
    }
    docker {
        docker.enabled          = true
        conda.enabled           = false
        singularity.enabled     = false
        podman.enabled          = false
        shifter.enabled         = false
        charliecloud.enabled    = false
        apptainer.enabled       = false
        docker.runOptions       = '-u $(id -u):$(id -g)'
    }
    singularity {
        singularity.enabled     = true
        singularity.autoMounts  = true
        conda.enabled           = false
        docker.enabled          = false
        podman.enabled          = false
        shifter.enabled         = false
        charliecloud.enabled    = false
        apptainer.enabled       = false
    }
    apptainer {
        apptainer.enabled       = true
        apptainer.autoMounts    = true
        conda.enabled           = false
        docker.enabled          = false
        singularity.enabled     = false
        podman.enabled          = false
        shifter.enabled         = false
        charliecloud.enabled    = false
    }
}

• test.sh

#!/usr/bin/env python3
from multiprocessing import Pool

def worker(num):
    return num * 2

if __name__ == '__main__':
    with Pool(processes=10) as pool:
        pool.map(worker, range(3 * int(1E8)))

I made on purpose a python script that should take more than 1GB, while I am allocating only 1GB to the process. Results: If I run test.nf with docker:

nextflow run <http://test.nf|test.nf> -c test.config -profile docker

the run fails with an OOM (error 137). If I run it with conda:

nextflow run <http://test.nf|test.nf> -c test.config -profile conda

it succeeds but it takes like 10GB of RAM (seen with htop). It's "normal" since you cannot basically limit resource allocation with a process launched through conda. Although ideally it's be really nice to able to do that. I would expect apptainer to behave similarly to docker, since it's a container engine. However, when I run test.nf with apptainer or singularity:

nextflow run <http://test.nf|test.nf> -c test.config -profile <apptainer / singularity>

the behaviour is the same as for conda!! It does not fail but consumes a lot of RAM to run the process. I saw the same problem but for the number of CPUs used by a process when I ran apptainer. Do you know what is happening? Should we add additional arguments when using apptainer / singularity? Thanks a lot

apptainer can not limit memory in all cases

Well, the thing is I don't want to take up all CPUs and memory available in a multi-user environment. If I specify N cpus and this amount of memory, I want my process to have at most those resources and not more. It's really an issue

but then just do

--memory ${task.memory}

in

apptainer.runOptions

?

Yeak ok, but why Nextflow does not do that in the first place? The expected behaviour would be to have automated resource limitations by Nextflow.

maybe even

--cpus ${task.cpus}

apptainer can also do that

Oh ok. In my opinion it should be written in other places too because it's very cumbersome. And anyway I don't think it's a good idea altogether

I think, generally nextflows behaviour is correct, the scheduler should be concerned with the task requirements, because it should use those for scheduling, offloading this to the container engine is more of a way to enforce limits, rather than improve scheduling

No I do not agree with that, I'm sorry. The resource limitations should be the same, whatever executor you use

Well, the linux executor simply does not enforce any limits

Whether you're running local tests or CI tests on Ubuntu runners, you're using the local executor. And if you want a process to have at most N cpus and M memory, it's just bad practice to relax those constraints when you're running tests, because then the environment will be different once you deploy

Oh yes..I had a similar thread couple of weeks ago: https://nfcore.slack.com/archives/C043FMKUNLB/p1762937074031609

I think this information is not accurate, and apptainer can also do this, but maybe it is not implemented nicely in the nextflow container driver

The purpose of CI testing (not only in bioinformatics) is to provide conditions as similar as possible as real life, once deployed. It is precisely the purpose of the test_full

Which is fine if you have a defined production environment, but thats kind of explicitly absent / not defined for nf-core modules and pipelines, and much of the effort is providing reproducibly across environments

In general, Nextflow cannot enforce resource limits because it usually delegates the task to a scheduler With the local executor, I suppose we could try to enforce these limits using OS primitives like

/proc

, but we never felt that it was worth doing. OOM killers are tricky to get right Also because you can just run the local task with Docker or Podman and get that behavior Looking at the Nextflow codebase, it seems DockerBuilder is adding the

--memory

option whereas SingularityBuilder / ApptainerBuilder are not So if it's as simple as adding

--memory

to the apptainer command generated by Nextflow, then let's do it

Adding optional

--memory

to apptainer would be good, but having it always would cause problems for me. I'm using unprivileged apptainer with slurm,. apptainer uses cgroups for memory which I cannot do

Feels like it wasn't long since this was up the last time. I don't remember exactly all the ways singularity/apptainer can work, but there used to be a lot of different things done to support various ways of working depending on what was available in the environment, and I'd certainly be wary for something such breaking in some environments. Coupled with how they are very often used in environments where there is a scheduler that enforces the limits, I'd say the case for adding these options is rather weak.

I see, in that case maybe we could add a flag like

executor.$apptainer.memoryLimits

to make it opt-in

I totally agree with you all, forcing a change would be a very bad idea and would break lots of existing things. But perhaps having opt-in parameters for memory and cpus would be good idea