process_gpu

withLabel: process_gpu {
    ext.use_gpu      = true
    accelerator      = 1
    containerOptions = {
        if (workflow.containerEngine in ['singularity', 'apptainer']) {
            return '--nv'
        }
        return '--gpus all'
    }
}

process_gpu

accelerator

ext.use_gpu

gpu

containerOptions

runOptions

--gpus all

docker run

task.accelerator

--num-gpus

1

gpu

process_gpu

--use_parabricks_star

You will need to add the accelerator directive to your local config in order to make use of the GPUs correctly if you do not use a cluster setup with a dedicated GPU queue. The accelerator diretive is added automatically for the executors

'awsbatch','google-batch','hq','k8s'

. See [Nextflow executors](https://www.nextflow.io/docs/latest/executor.html) for more infos.

accelerator

CUDA_VISIBLE_DEVICES

accelerator

--num-gpus

task.ext.num_gpus

because omitting

accelerator

on local/SLURM also drops

--num-gpus

from the tool command.

--gpus all

--gpus $num

--gpus '"device=1,2"'

Number of GPUs to use for a run. (default: 1)

containerOptions = {                                                       
      if (workflow.profile.contains('gpu')) {               
          if (workflow.containerEngine in ['singularity', 'apptainer']) {
              return '--nv'
          }
          def ngpus = task.accelerator?.request ?: 1
          return "--gpus ${ngpus}"
      }
      return null
  }

--gpus null

Won't that be

--gpus null

? in certain situations?

process.containerOptions

params.gpu_container_options = "--gpus 1"

withLabel: process_gpu {
    containerOptions = params.gpu_container_options
}

process_gpu

containerOptions

withLabel: process_gpu {
    ext.use_gpu      = true
    accelerator      = 1
    containerOptions = {
        if (workflow.profile.contains('gpu')) {
            if (workflow.containerEngine in ['singularity', 'apptainer']) {
                return '--nv'
            }
            return params.gpu_container_options ?: '--gpus all'
        }
        return null
    }
}

gpu

--gpu_container_options '--gpus 1'

accelerator = 1

// before
def num_gpus = task.accelerator ? "--num-gpus ${task.accelerator.request}" : ''
// after
def num_gpus = task.ext.num_gpus ? "--num-gpus ${task.ext.num_gpus}" : ''

accelerator

nextflow.config

Does this look right?

return params.gpu_container_options ?: '--gpus all'

params.gpu_container_options

--gpus

--use-gpu

-profile gpu

--gpu

--gpu

-profile gpu

queue

gpu

Thanks for the fix!

Just a matter of preference, and again not that familiar with nextflow yet so I am probably missing context, but I think requiring a flag --gpu for only some configurations confusing. I would think that saying "use the GPU aligner" or "this step uses a GPU" would be enough signal from the user.

I am also a little confused on how this will effect specifying the number of GPUs I want a task to use. I had thought that passing an accelerator requirement would have communicated to slurm to pass a GPU resource requirement, but I'm not sure that is happening. I'm going to play around with this next week and see how this works, since the parabricks team had recommended going with two GPUs per alignment task.

containerOptions

--use-gpu

containerOptions

process_gpu

--gpus all

--nv

--use_gpu

--use_gpu

--use_parabricks_star

I would think that saying "use the GPU aligner" or "this step uses a GPU" would be enough signal from the user.

--tools

--skip-tools

--tools parabricks

--tools parabricks_deepvariant

run_whatever

skip_whatever

params

Note: For faster CPU runs with BWA-Meth, enable the BWA-MEM2 algorithm using --use_mem2. The GPU pathway (Parabricks) requires -profile gpu and a container runtime (Docker, Singularity, or Podman); Conda/Mamba are not supported for the GPU module.

params.gpus_per_parabricks = 2
accelerator = params.gpus_per_parabricks
ext.args = "--num-gpus ${gpus_per_parabricks}"

task.cpus

ext

def num_gpus = task.accelerator.request ? "--num-gpus ${task.accelerator.request}" : '' // or --num-gpus 1 as default?

task.ext.num_gpus

task.accelerator.request

def num_gpus = task.accelerator?.request ? "--num-gpus ${task.accelerator.request}" : '--num-gpus 1'

setting the accelerator lead to runs breaking on my cluster which is why we only set it for specific executors

--gpus all

--nv

accelerator

task.accelerator.request

--num-gpus

containerOptions

process_gpu

accelerator = 1

process_gpu

--use_gpu

--use_parabricks_star

ext

task.accelerator.request

--num-gpus

accelerator = 2

task.accelerator.request

--num-gpus 2

--gpus all

accelerator = null

withName

Only needed on local/HPC. Cloud executors (AWS Batch, K8s etc.) handle this via their own APIs.

process {
    withLabel: 'GPU' {
        accelerator = 1
        containerOptions = { params.gpu_container_options ?: (workflow.containerEngine in ['singularity', 'apptainer'] ? '--nv' : '--gpus all') }                 
    }
    withName: 'PARABRICKS_.*' {
        ext.args = { 
            [
                task.accelerator?.request ? "--num-gpus ${task.accelerator.request}" : '--num-gpus 1',
                // Parabricks args
            ].join().strip()
        }
    }
}

acclerator

docker exec

accelerator

withName: 'PARABRICKS_.*' {
        ext.args = { 
            [
                task.accelerator?.request ? "--num-gpus ${task.accelerator.request}" : '--num-gpus 1',
                // Parabricks args
            ].join().strip()
        }
    }

def num_gpus = task.accelerator ? "--num-gpus ${task.accelerator.request}" : ''

args

arg

1.2 Non-file mandatory command arguments

Non-file mandatory arguments or arguments needed to modify the command to make the module run with no error, SHOULD be provided as value channels (for example

lib_type

in salmon/quant) - see ‘Input/output options’ below.

1.3 Optional command arguments

All non-mandatory command-line tool non-file arguments MUST be provided as a string via the

$task.ext.args

variable.

--threads ${task.cpus}