What's the best practice for GPU modules? I saw nf-co.re/docs/developing/components/gpu-modules , but I'm wondering who and where

process.accelerator

(docs.seqera.io/nextflow/reference/process#…) should be set by. Does every module that checks

task.accelerator

have to have the

process_gpu

label? Not all currently do ( which means they default to cpu - why not default to gpu? ) Should pipelines need to set

process.accelerator

? The

-profile gpu

sets

task.accelerator = 1

but I guess this should be overridden by an actual map that assigns

type

and

request

. It would make sense that institutional configs set this - but if they support multiple types, how exactly should this be supplied by the user? Is a user expected to override settings with a custom config with process selectors and set the accelerator?

process {
    withName: 'GPU_PROCESS_ONE' {
        accelerator = [ type: 'h100', request: 4 ]
    }
    withName: 'GPU_PROCESS_TWO' {
        accelerator = [ type: 'l40s', request: 1 ]
    }
}

and then if they're using an executor that doesn't directly support the directive, should the institute config add it to say

process.clusterOptions

via the cluster directives e.g. slurm -> grus Is it turned off for a particular module in the same way

process {
    withName: 'GPU_PROCESS_ONE' {
        accelerator = null
    }
}

and if so, what's the use of

ext.use_gpu

now? And if overriding generally, I guess they should first override

withLabel: 'process_gpu'

and then use

withName

?

(This is related to a institutional config PR, but touches on other issues.)

I made an attempt to update the docs with the relevant information: github.com/nf-core/website/pull/4352

Maybe this PR / discussion is also related: github.com/nf-core/sarek/pull/2181

Thanks. It seems that PR at least agrees with my understanding. If Sarek continued to do it that way, a user would manually need to override the accelerator in order to use it on say slurm or locally. The way Paolo suggests is what I would expect.

Can you tell me why setting it per standard to 4 does not have that problem then?

Sorry, which problem? The

accelerator

directive does nothing by itself unless it's one of those four executors. However, having it set allows other executors to use it to determine if they should enable their GPU settings.

Because I am eager to adapt and just denote in the docs what people need to do in their local configs. My goal was to make it as easy to use as possible and that seemed the best way but I can see the issues that paolo pointed out so I am happy to change.

I can have a go; because as Paolo mentions - it breaks consistency meaning if it's run on kubernetes it would run with

accelerator=4

whereas using the

slurm

executor in a cluster with the same GPUs available would have it run without.

This is what I was trying to clear up in my PR, and how it flows through the different configs

I ran into problems when it was set to 4 on my hpc because we have a dedicated gpu queue (see github.com/nf-core/configs/blob/…/m3c.config). This is all a bit confusing for me to be honest because there is so many settings to set at different places 😄

This was what we did to redirect processes to our gpu queue/partition. github.com/nf-core/configs/blob/…/pelle.config#… The idea is to check if

task.accelerator

is set by the pipeline ( either through

-profile gpu

or

--enable_tool

). Generally at this point, only the

task.accelerator.request

is set, and then it's down to the cluster to set the

task.accelerator.type

and redirect it to the correct queue.

Can I ask why you would not utilize the label of the process?

Because the user might want to turn it off

There's no guaranteed consistency for that such that an institutional config can rely on it.

That too. Ribodetector is an example

ok so similar to task.time which we use to understand if we need a long-running node we then use task.accelerator to understand whether we need the gpu or not -> got it! I will try and adapt both the sarek config (or rather accept paolos PR) and my own Thanks!