Hi all, I can't find the way to limit the number of jobs that my pipelines are throwing on my slurm cluster... The command looks like :

nextflow run nf-core/rnafusion  \
   -profile singularity \
   -c cluster.config \
   --input rnafusion_sample_sheet.csv \
   --all \
   --outdir 06rnafusion \
   --genomes_base rnafusion_references \
   -resume

And my

cluster.config

:

singularity {
    enabled    = true
}


process {
    resourceLimits = [
        memory: 252.GB,
        cpus: 56,
        time: 96.h
    ]
    executor = 'slurm'
    queueSize = 5
    clusterOptions = { "-A myaccount"}
    queue    = { task.time <= 24.h ? 'short' : 'long' }
}

params {
    igenomes_ignore = true
    max_memory      = 252.GB
    max_cpus        = 56
    max_time        = 96.h
}

If some one has an idea of what I am doing wrong, I would greatly appriciate ! Benoit

queueSize is supposed to be inside the executor clause. Slurm profile example:

profiles {
    slurm {
        executor {
            name               = "slurm"
            queueSize          = 30
            queueGlobalStatus  = true
        }
    }
}

Amazing, many thanks! When I was checking different profiles in the git, it was unclear why some limitations are given

process

chunk and other in the

params

chunk. In my case, I would like to make sure the sum of all my jobs requirements do not exceed 1TB RAM and 192cpus. Is it possible ?

Good question. I think that's up to the system/node limits (and/or scheduler). If there are available resources, the system will give it to you. Another thing you can do, again on the process level though, is cap the maximum resources, per process, like this:

process {
    resourceLimits = [
        cpus: 4,
        memory: '15.GB',
        time: '1.h'
    ]
}

This will save you from errors, in cases for example, when your module requires more memory than the system can provide (used mainly for tests in local machines).