Hi all 🙂 I am trying to create a config for the HPC of my institute, and I am wondering if you could give me some insights, being quite a beginner. The cluster, has a number of different partitions, Slurm as a scheduler, and a default partition

gp

for jobs to run when no partition is specified. This means that anyone running nextflow is flooding the

gp

partition, creating a huge queue on this partition. Is there a way to allow nextflow assign the main and each task to different partitions based on availability? From what I understand, the main bash script, where I run the nextflow head process with

#SBATCH --partition=p1,p2

will NOT inherit the same

#SBATCH

options to its children and let slurm do the partition selection for each task? I will have to create something custom in the config, right? Does it worth it to further investigate this, or it is not in the context of institutional configs?

That seems to be in the context of institutional configs. The config

pdc_kth

has something similar: https://github.com/nf-core/configs/blob/master/conf/pdc_kth.config where there are several partitions,

shared

,

memory

,

main

, etc. To future proof a bit, I wouldn't def the function as it is in the config above because they're not going to be supported in future, but you can still put the function in the closure:

process {
    resourceLimits = [
        memory: 1790.GB,
        cpus: 256,
        time: 7.d
    ]

    executor = 'slurm'
    clusterOptions = {
        def selectPartition = { cpus, memory, time ->
            def slurmOpts = [
                "-A ${params.project}",
                params.clusterOptions?: "",
                time <= 7.d && mem <= 111.GB && cpus <= 256 ? '-p shared' : '',
                ... 
                <more partition selection options>
            ].minus("").join(" ")
        // Call function based on resources allocated
        selectPartition(task.cpus, task.memory, task.time)
    }
}

And it's correct that there's no inheritance/picking up stuff from the job (if any) where the main nextflow monitoring process runs. I'm not sure if I understood if this is a single system with many partitions or many systems, but each "system" needs its own

resourceLimits

to tell nextflow how much e.g. memory is available. On top of that, it's typically a good idea to have

process.queue

be a closure (code bit) that uses needed bits from

task

(e.g.

memory

and

cpus

) to calculate the queue (which will be passed after

-p

in the sbatch syntax. Typically, a HPC profile will also signal use of apptainer/singularity (where using "singularity" in nextflow is recommended for historic reasons, no matter if you have singularity or apptainer). If more adjustments are needed you can work with

clusterOptions

but I generally recommend trying to respect anything the user sets there to fix things.

sounds great, thanks a lot for your suggestions, and pointing me to these resources!! indeed there are different resourceLimits for each partition that I have to define. I can go your way and pick the lowest, but if I want to define all of them, do you have an idea on how could I do this?

For partitions you don't, you define it once and pick the fitting partition(s) from the values for the task.

ok, I see. thanks for your help!