Hi guys, I am working on a new version for this config as it is quite out of date. I am working on a version where I want the users to be able to switch specs according to the partition they are using and the account (maybe trying to be too fancy here? 😄 ). I have been looking into different config files and with the help of AI I got to the version below. However, there are a few things here that I am not sure because I don’t have enough experience. 1) AI added some safeguards that I think are nice but not sure how good practice it is; 2) as I am trying to add options for the different partition there are things such as

Math.min(p.max_cpus as int, s.max_total_cpus as int)

being computed, which again I am unsure if this is the best way; 3) I am not even sure if inferring things from a parameter such as

project

(I am trying to infer the priority level of the account because then the specs change) is good practice at all. Many thanks.

/*
 * Cambridge CSD3 institutional profile
 *
 * Partitions:
 *   - icelake        : 76 CPUs, 256 GiB RAM
 *   - icelake-himem  : 76 CPUs, 512 GiB RAM
 *   - sapphire       : 112 CPUs, 512 GB RAM
 *
 * Service levels:
 *   - inferred from params.project (e.g. MYGROUP-SL2-CPU)
 *   - params.service_level can override
 *   - sl1/sl2 -> 36h
 *   - sl3     -> 12h
 */

// ---- Resolve partition ----
def partition_specs = [
    'icelake'       : [ max_cpus: 76,  max_memory: 256.GB ],
    'icelake-himem' : [ max_cpus: 76,  max_memory: 512.GB ],
    'sapphire'      : [ max_cpus: 112, max_memory: 512.GB ]
]

def selected_partition = params.partition ?: 'icelake'
def p = partition_specs[selected_partition]

if (!p) {
    throw new IllegalArgumentException(
        "Unsupported partition '${selected_partition}'. " +
        "Supported: ${partition_specs.keySet().join(', ')}"
    )
}

// ---- Infer service level from project ----
def service_specs = [
    'sl1' : [ max_time: 36.h, max_total_cpus: 4256 ],
    'sl2' : [ max_time: 36.h, max_total_cpus: 4256 ],
    'sl3' : [ max_time: 12.h, max_total_cpus: 448  ]
]

def inferred_service = 'sl2'
if (params.project) {
    def project_upper = params.project.toUpperCase()
    if      (project_upper.contains('-SL3-')) inferred_service = 'sl3'
    else if (project_upper.contains('-SL2-')) inferred_service = 'sl2'
    else if (project_upper.contains('-SL1-')) inferred_service = 'sl1'
}

// Manual override wins
def selected_service = (params.service_level ?: inferred_service).toLowerCase()
def s = service_specs[selected_service]

if (!s) {
    throw new IllegalArgumentException(
        "Unsupported service_level '${selected_service}'. " +
        "Supported: ${service_specs.keySet().join(', ')}"
    )
}

// ---- Params ----
params {
    config_profile_description = 'Cambridge HPC CSD3 cluster profile.'
    config_profile_contact     = 'Raquel Manzano (<mailto:rm889@cam.ac.uk|rm889@cam.ac.uk>) and Andries van Tonder (<mailto:ajv37@cam.ac.uk|ajv37@cam.ac.uk>)'
    config_profile_url         = '<https://docs.hpc.cam.ac.uk/hpc>'

    partition     = selected_partition
    project       = null
    service_level = selected_service

    max_memory = p.max_memory
    max_cpus   = Math.min(p.max_cpus as int, s.max_total_cpus as int)
    max_time   = s.max_time
}

// ---- Singularity options ----
singularity {
    enabled    = true
    autoMounts = true
}

// ---- JAVA limits for nextflow ----
// Note that we recommend running nextflow with `srun` + screen/tmux
// or via `sbatch` for large or long-running workflows
env {
    NXF_JVM_ARGS = '-Xms2g -Xmx4g'
}

process {
    // Hard caps from selected partition + service level
    resourceLimits = [
        memory: params.max_memory,
        cpus:   params.max_cpus,
        time:   params.max_time
    ]

    // Module commands provided by HPC
    beforeScript = '''
    . /etc/profile.d/modules.sh
    module purge
    module load rhel8/default-icl
    '''

    executor = 'slurm'
    queue    = params.partition

    // Only pass account; queue already defines partition
    clusterOptions = params.project ? "-A ${params.project}" : ''

    cache   = 'lenient'
    scratch = true

    errorStrategy = { task.exitStatus in [143, 137, 104, 134, 139, 140] ? 'retry' : 'finish' }
    maxErrors     = '-1'
    maxRetries    = 2

    withLabel: process_single {
        cpus   = { 1 * task.attempt }
        memory = { 4.GB * task.attempt }
        time   = { 4.h * task.attempt }
    }

    withLabel: process_low {
        cpus   = { 2 * task.attempt }
        memory = { 8.GB * task.attempt }
        time   = { 8.h * task.attempt }
    }

    withLabel: process_medium {
        cpus   = { 4 * task.attempt }
        memory = { 16.GB * task.attempt }
        time   = { 12.h * task.attempt }
    }

    withLabel: process_high {
        cpus   = { 8 * task.attempt }
        memory = { 32.GB * task.attempt }
        time   = { 24.h * task.attempt }
    }

    withLabel: process_long {
        time = { 36.h }
    }

    withLabel: process_high_memory {
        memory = { 128.GB * task.attempt }
    }
}

// ---- Executor tuning: necessary to not overload SLURM ----
executor {
    name              = 'slurm'
    queueSize         = 200
    pollInterval      = '5 min'
    queueStatInterval = '5 min'
    submitRateLimit   = '10 sec'
    exitReadTimeout   = '30 min'
}

Any chance you can get the partition via hostname?

Are there any rules about who can use what partition? Otherwise, why should one have to care about that as a user?

Try running:

nextflow lint conf/cambridge.config
nextflow run <http://configtest.nf|configtest.nf> -profile cambridge

with the latest edge nextflow to be sure

I don't understand what the bits about service_level/specs et.c. is supposed to do.

Yeah just to reiterate on the above: all the labels specificaltions should be removed - they already exist in the pipelines themselves, they should only be used in pipeline-specific institutioanl configs. But given partitions are just queues, I don't think you need to update the

max_resources

each time. You just need to set the maximum limits of the biggest parition, then select teh queue depending on the specific task resource requests, I would assume (even if seems the requests are simply just memoryu/cpus, no fancy GPU stuff)

Awesome guys, thank you very much for the input. Regarding the partition, each partition have different specs (cpu and mem), that’s why I thought it would be nice to be able to change the max cpu and mem accordingly. I know slurm allows to submit to different partition but to be honest that does not seem something that our HPC admins support at the moment. You could get the partition via

hostname

in principle but would involve some lines of code which I am not sure if the config syntax would be happy with. If I set the config to the partition with the biggest resources, woudn’t I create the risk of a job asking for too much mem/cpu in a partition with less resources?

f I set the config to the partition with the biggest resources, woudn’t I create the risk of a job asking for too much mem/cpu in a partition with less resources?

No, because the other part I mentioned that the

queue

closure should always make sure it goes to the right partition (unless I have missed a more complicated detail about the way the cluster works)

Oh, do you mean if the queue itself will adjust an user’s request?

Yes 🙂

It will just throw an error

Oh, so you can't do like

queue = { task.cpu ? 48 & task.memory > 128 ? 'partition1' : 'partition2'  }

?

"something that our HPC admins support at the moment" - do you mean "like" or "support"? This is core slurm functionality. It can of course break if they do special things (e.g. lua submit), but that's rather rare.

Real example from sanger: https://github.com/nf-core/configs/blob/0bd43c1ad6c29c44b2c1320bbd8e2e387458c47c/conf/sanger.config#L22

Nice! yeah, I can try to implement something like this

I would recommend not forcing users to make choices unless that's needed.

@pontus I believe it is something they do not support. I tried it (years ago) and they told me that it woudn’t work and since then I only submit to one partition. I can check with them again if that has changed

You can just try?

But if I try most of the jobs will go to the biggest partition and not sure if any will go to the other one 🤔

*   - icelake        : 76 CPUs, 256 GiB RAM
 *   - icelake-himem  : 76 CPUs, 512 GiB RAM
 *   - sapphire       : 112 CPUs, 512 GB RAM

Do you mean here that icelake and icelake-himem are pretty much the same except for the himem bit?

My point is that unless (for some specific reason that's hard to imagine) they don't like jobs queuing for multiple partitions, you can just user all relevant and it will start in the first available, so testing would mostly be to see that it doesn't break when submitted.

SLURM takes a comma separated list for the

--partition

parameters, so you should be able to set

queue = "icelake,icelake-himem,sapphire"

and thus accomplish what the sysadmin probably should have done in the first place -- make all nodes available to any given process and let SLURM pick. The point of partitions is not to group various nodes (as seems to be the attempt here), but to group classes of processes, e.g. low-resource & quick, low-resource and long-running, high-resource & quick, etc. The idea is that if you pick the correct queue, you can get a quick, low-resource job in, even if there are a bunch of jobs ahead of you, as long as they need more resources than that queue provides. BTW, a node can be used in several queues.

But if I try most of the jobs will go to the biggest partition and not sure if any will go to the other one 🤔

SLURM will see if each partition allows the request of the job, and among those that do it will pick the one where the job would start soonest. If one partition fills up, another will become more favorable to the scheduler. (As mentioned, in a more well thought-out setup, you wouldn't have so many queues that all would allow a job.) For the

resourceLimits

you just pick the maximum of any partition. The only thing you have to be careful about here is that you don't allow the pipeline to make a request that no partition can fulfill (e.g. if you have one with lots of RAM and another with long runtimes, but none with both, you'll have to decide between them.) Not sure you have such a scenario here, but if you do, you can redefine the relevant slot (

time

,

memory

,

cpus

) in

resourceLimits

from some baseline to something higher (if necessary while limiting another slot) based on e.g. process label. I am not sure what the whole "service level" thing is about -- maybe I'm blind (it happens occasionally), but I could not find where that this actually used to influence SLURM in any way. Is this meant to be the

--qos

parameter of

sbatch

? If so, you would have to set that in

clusterOptions

. E.g., I have the following in my config:

process {
    executor       = 'slurm'
    clusterOptions = { task.memory <= 240.GB ? (task.time > 24.h ? '--qos=long'
                                                                 : '--qos=normal')
                                             : '--qos=mem' }
    // other stuff here ...
}

(I'm too lazy to translate your situation, but I think you get the point.) Lastly, I have some concern about using

params

. The config per-se does not care if a parameter exist or is possible, but the pipeline you are running might, and if it validates all parameters and crashes upon encountering one it does not recognize (e.g. because that one was specifically designed to be only used in the config), that's a problem, because you can't use that parameter with that pipeline. (The other thing I don't like is that any

help

function of the pipeline would not know about config

params

, and that makes it harder for the user to even figure out that they exist and when they should be used.) If you really need user input to decide something, I would just make different configs to begin with, and user can pick the correct one via

-profile

. I'm not sure why you would need this here (e.g. if the user really wanted or needed to override the

queue

or

--qos

within

clusterOptions

, that could be done in a custom config, but ideally the config should pick the appropriate setting based on the requested resources). In short, your config may (and possibly should) set

params

(e.g.

config_profile_description

& friends, for really old nf-core pipeline versions also

params.max_memory

& Co.), but it should never expect and read

params

. E.g., I find it very elegant that you are using

parmas.max_*

to set the

resourceLimits

, because one of those will work with a (nf-core) pipeline of pretty much any age, but unfortunately it might crash newer (or non-nf-core) pipelines when you set a parameter (like

--max_memory

) they don't recognize. (Many pipelines ignore unknown parameters or just issue a warning, but there is no guarantee that all of them will, and ideally your

-profile

is not limited to nf-core but works for all Nextflow pipelines.)

Thanks everyone. This week I am unable to work on the config but will get back to it asap. Your comments are very useful, I will gather them and get in touch with the cluster admins to discuss a few things to cover some of your points.

If they have such restrictions set, I guess it's up to local knowledge to know how they're used. Is

icelake

available to all?

Yes

I would agree then to just set icelake as a default 🙂

Okay, yeah, that will probably make things easier to understand. Then my other question is related to the account level (SL3/SL2). SL3 is limited in memory and cpus but free, but most people have an sl2 to work with. Shall I assume SL2 in the config for simplification as well?

Yes I think that's fine

Possibly depends on how easy it is to check and what the various limits are.

The limits are in the docs, but after seeing what the ai is spitting out as config I am not so sure it is straight forward

The AI may be fond of overly complex and suboptimal solutions.

Yeah, that’s true, so maybe a simpler solution would be what James recommended already that they did at Sanger: https://github.com/nf-core/configs/blob/0bd43c1ad6c29c44b2c1320bbd8e2e387458c47c/conf/sanger.config#L22

Not sure I understand, but if the setup isn't totally weird, the servicelevel (if it decide things like how large memory a job can request) would be in

resourceLimits

.

oh yeah, absolutely! Would it be okay to write a resourcelimit with conditional depending on the service level then? That might require another param from the user… but they should set up their account anyway

@Raquel Manzano I'm still sort of confused what the idea between the different partitions are, do the sys admins provide guidance somewhere of when to use which?

Yes, having that conditional sounds sensible and will probably look something like

process {
  resourceLimits = [
    time: {some code}.call(),
    cpus: maybeisstaticsonocode,
    memory: {some other code maybe}.call()
  ]
}

You already have the

resourceLimits

determined (in part) based on the "service level" in the draft config. It's a bit convoluted, but it should work, except my concerns about

params

. What the defaults should be (for partition and SL) is something to discuss with the sysadmins. I would be inclined to go with the most basic and let users ask for more if they need. Regarding the use of

params

, as mentioned I don't like using that for configs, because if a pipeline validates

params

and croaks on ones that it does not recognize, that's a deal breaker. I would carefully think about that and maybe do some testing. But if you don't use

params

, how else can you capture user input? I see these possibilities: • Set an environment variable and check that in the config. • Have a different config for e.g. each SL and/or partition, so you would chose a different

-profile

. With lots of combinations, this may not be very practical, though. If the sysadmins are helpful, they could put the "correct" setting of the environment variable(s) into each user's

.bashrc

(or somewhere equivalent) -- of course power users can override that. My best guess about the partitions: They bought over the years different sets of nodes, and each set is basically its own cluster, except that all of them are accessible via the same head node(s). Maybe there are good technical reasons to keep them separate, but to me that seems like a questionable decision and I would have combined them into one partition. If you set one as default, that one will be hammered, while the others will be underutilized. But that's not you problem -- the sysadmins have to figure that out. If they are open to discussion, I would talk to them and try to accommodate their suggestions.

Thomas you are on point with your explanation about the partitions, there are no guidelines as to when to use what. You sort of know that if you want more memory you can use

sapphire

, if you want even more then have to go for the

himem

partitions but that is basically it, you have to play around with the queue depending on how much is being used.

icelake

is the biggest one and that’s normally the preferred choice but

sapphire

even though is smaller the jobs run faster because is newer but then you might have to wait longer for your jobs to start. That’s why is basically up to the user and the reason why I want to give a choice (exactly what Thomas said, I don’t want icelake queue being overflowed with nextflow jobs).

Ok thank you very much for the explanation, then I guess my vote would then indeed be a combination of what you/Pontus suggested would be the best way forward with an extra parameter for users (ignored in parameter validation, mind) to select which one then dynamically pick the resource usage accordingly

Can you guarantee that a parameter will be ignored by any pipeline you chose to run? (Can that be done in a config?) Because if not, an environment variable may be preferable.

You can tell nf-schema to ignore it from the config. You of course can't be certain that any pipeline doesn't start to use your special "secret" variable, but that goes for environment variables as well.

Mmm maybe then, I will set as default icelake but allow an environmental variable to change to other partitions?

I don't think it's a problem using a param, you can have that with a fallback if it's not set.

For the service level, max cpus, mem and time changes. But I should be able to infer it from the account name as all of the are something like

groupname-sl2-something

(And it's mostly those that know the user base that can say what makes sense.)

Alright, a

param

it is for the partition

Less configuration is generally better.

Okay, new version below. I run the

nextflow lint

and

nextflow run <http://configtest.nf|configtest.nf>

and all was okay.

// nf-core/configs: Cambridge CSD3 cluster profile
// Available partitions:
//   - icelake        : 76 CPUs, 256 GB RAM
//   - icelake-himem  : 76 CPUs, 512 GB RAM
//   - sapphire       : 112 CPUs, 512 GB RAM
//
// The profile defaults to the broadly available `icelake` partition but allows
// users to override it with `--partition`. Walltime is inferred from the SLURM
// account name when possible: projects containing `-SL3-` use a 12 h cap,
// otherwise the profile assumes the common SL1/SL2 36 h limit.

params {
    config_profile_description = 'Cambridge HPC CSD3 cluster profile.'
    config_profile_contact     = 'Raquel Manzano (<mailto:rm889@cam.ac.uk|rm889@cam.ac.uk>) and Andries van Tonder (<mailto:ajv37@cam.ac.uk|ajv37@cam.ac.uk>)'
    config_profile_url         = '<https://docs.hpc.cam.ac.uk/hpc>'

    partition = 'icelake'
    project   = null

    // Compatibility with nf-core schema validation across pipeline versions.
    schema_ignore_params         = 'partition,project,max_memory,max_cpus,max_time,validationSchemaIgnoreParams'
    validationSchemaIgnoreParams = 'partition,project,max_memory,max_cpus,max_time,schema_ignore_params,validationSchemaIgnoreParams'
}

validation {
    ignoreParams = ['partition', 'project', 'max_memory', 'max_cpus', 'max_time', 'schema_ignore_params', 'validationSchemaIgnoreParams']
}

params.max_memory = {
    if (params.partition == 'icelake') {
        return 256.GB  // this is a whole node
    }
    if (params.partition == 'icelake-himem' || params.partition == 'sapphire') {
        return 512.GB // this is a whole node
    }

    System.err.println("ERROR: cambridge params.partition must be one of 'icelake', 'icelake-himem', or 'sapphire' (got '${params.partition}').")
    System.exit(1)
}.call()

params.max_cpus = {
    if (params.partition == 'sapphire') {
        return 112
    }
    if (params.partition == 'icelake' || params.partition == 'icelake-himem') {
        return 76
    }

    System.err.println("ERROR: cambridge params.partition must be one of 'icelake', 'icelake-himem', or 'sapphire' (got '${params.partition}').")
    System.exit(1)
}.call()

params.max_time = {
    def project_name = params.project?.toUpperCase()
    return project_name?.contains('-SL3-') ? 12.h : 36.h
}.call()

singularity {
    enabled    = true
    autoMounts = true
}

process {
    resourceLimits = [
        memory: params.max_memory,
        cpus: params.max_cpus,
        time: params.max_time
    ]

    executor       = 'slurm'
    queue          = params.partition
    clusterOptions = { params.project ? "--account=${params.project}" : '' }
}

I think this looks OK except for the duiplicated validationIGnore bits(?) But at this phase a PR would be more easy to leave commetns on @Raquel Manzano 🙂

Yes, I was thinking the same James. I am currently working on my fork and updating the md to do a proper PR

I agree this is probably ready for a PR, and is probably okay. I haven't written it out, but I think it could be rather beautified by putting the partitions in a map (in a paramsvar)

do you have any examples of that?

Something like (untested):

params.csd_time = {  params.project?.toUpperCase().contains('-SL3-')? 12.h : 36.h }.call()
 params.csd_parts = [
  icelake:  [ memory: 256, cpus: 76, time: params.csd_time], 
  "icelake-himem":  [ memory: 512, cpus: 76, time: params.csd_time], 
  sapphire: [ memory: 512, cpus: 112, time: params.csd_time], 

process {
    resourceLimits = params.csd_parts[params.partition]

    executor       = 'slurm'
    queue          = params.partition
    clusterOptions = { params.project ? "--account=${params.project}" : '' }.call()

Live now: https://github.com/nf-core/configs/pull/1102

To me that's easier to read, but I think either is fine (assuming they work) and you're the one maintaining :)

Yes, I will maintain this one for now

yes that's expected at the moment 🙂

For the moment, unfortunately, yes. But it shouldn't be later on.

ah, awesome

Btw don't forget to check your config match the SHINY NEW CONFIG SPECIFICATIONS: https://nf-co.re/docs/specifications/configs/overview

Is it this one? https://nf-co.re/docs/developing/migration-guides/config-strict-syntax/ link is not working

YEs, there are 4 pages after the overview page 🙂

btw, when I was doing a local

pre-commit run

I had to change this hook to

mszostok/codeowners-validator:v0.7.4

. I am not sure if this is because I am doing it locally so I did not push this change but letting you know just in case

@mashehu is this related to the in progress tools release?

nope, that's nothing to do with us i think

REgardless @Raquel Manzano can just let nf-core bot fix any issues on the issue itself 🙂

I did the

NXF_SYNTAX_PARSER=v2 nextflow lint

and all seems okay and went through the new syntax docs. I think I cover everything but let me know if you notice anything. I also sent the config to the hpc admins for their feedback. If everyone happy I will do the merge