Amazing, let me know if you run into any problems @Mathys! A general comment again because multiple people have pinged me about this: Docker and Apptainer don't work on the login node or when running Nextflow pipelines with the local executor. You HAVE to submit your nextflow pipelines to the slurm scheduler and worker nodes using the Slurm executor 😉

Fantastic thank for the heads up !

⚠️ Important message for using the cluster. Make sure you modify permissions on folders you want others to be able to write to. I just updated the singularity_cache folder in

/mnt/shared/singularity_cache

with

chmod -R 777 /mnt/shared/singularity_cache

to make sure everyone can write their singularity images there (wasn't before, my bad, sorry!). If you only want read permissions, then the default permissions should be fine! ⚠️

<!channel> Hey GPU folks, for anyone that did some work with the cluster related to GPU work (even if it was just setting up resources, as we have been doing for #C0351A8S487, could you add an update here so I can add it to the AMER slides keanu thanks

Oh I didn’t see this ! Will do so today !

<!channel> Good 2nd hackathon day to all of you! How is everything going with the Slurm cluster and GPUs here? Did anyone manage to run anything heavy on the GPUs? Please be sure to include any progress or updates enabled by GPU usage in the daily wrap up slides, as it helps us to justify the sponsoring and hopefully continue to get support like this in the future gratitude thank you

Hey @Florian Wuennemann! how should we login to graphana ?

So we are running today for the first time 🎉 tomorrow we will have, I think, a better stress test!

<!channel> Thank you everyone that accessed and utilized the GPU Slurm cluster provided by Nebius throughout this hackathon. IF you have feedback for the utilization or questions about it going forward, please feel free to contact @Alexander Kim or myself via DM or post your feedback here. ⚠️ Also, remember to move your scripts and data off the cluster by tomorrow evening Eastern time, because the cluster will be taken offline and all your data will be lost in the process! I will send another reminder tomorrow a few hours before the shutdown! nf core bounce nf core pet nf core infra nf core nf core zoom

@Mathys if you want to test the gpu metrics, you can

git clone

this branch https://github.com/FloWuenne/nextflow/tree/gpu-metrics-record and execute a pipeline. You can then get the metrics from the tracelog via a variation of this command:

trace {
    enabled = true
    file = "${projectDir}/results/execution_trace.txt"
    overwrite = true
    fields = 'task_id,name,cpu_model,status,exit,realtime,%cpu,rss,gpu_model,gpu_mem_total,gpu_driver,avg_gpu_util,avg_gpu_mem_perc'
}

I will rename the field names tomorrow, but this should work for now!

amazing !

I will try and let you know how it goes !

⚠️ <!channel> Reminder to move your data off the Slurm cluster today! You are still free to use the cluster today until 5pm ET, when it will be shutoff and all associated folders, scripts and data will be lost forever. So make sure to copy any valuable data you processed and push your code to github! ⚠️

Is it premature to add a

withLabel:'process_gpu'

block to a public config file (E.g. the one I help maintain at https://github.com/nf-core/configs/blob/master/conf/seattlechildrens.config)? I have already used the GPU with Nextflow on some of our internal custom pipelines, so I know what I need to put in there to get it working on our cluster. I see at least that nf-core/scdownstream uses this label, and I can imagine some of our cluster users wanting to use that pipeline.

I started on GPU support in nf-core/modules for deepvariant (the google one, not parabricks) https://github.com/nf-core/modules/pull/8886

I'm not sure what I'm doing though 😉 And I don't have access to NVIDIA hardware locally, only Apple M1/M3, so I can't run the docker tests locally

I got a run to complete (though as expected the checksums are off) but don't know if GPU was used or not https://github.com/nf-core/modules/actions/runs/16886736125/job/47836374260?pr=8886#step:6:587

@Marcel Ribeiro-Dantas has some issues with running GPU tests, anyone around to comment on this PR?: https://github.com/nf-core/modules/pull/9330

Might throw a bottle at sea with this, but do we know if people are re-implementing bioinfo algorithms in CUDA/Triton ?

for instance, someone is asking me if VeryFastTree or iqTree2 is going to see CUDA support

In general, I feel like many bioinfo algorithms can be gpu accelerated

Hey @Mathys, so maybe not directly answering your question but NVIDIA is optimizing a few drug design tools for even faster GPU execution using something they call NIMS: https://build.nvidia.com/models?filters=usecase%3Ausecase_drug_discovery. @Edmund and I are looking at how this can be run from within Nextflow the most efficiently at the moment. I agree that there is likely a lot of tools that can be GPU optimized. From what I have seen, a maybe even bigger trend seems to be rewriting old, crusty bioinfo tools in rust for better and faster execution. Would be cool to have a catalog of examples for old tools that were rewritten for GPU / Rust / something else efficient.

Yup I think it would be quite valuable

Hi all, I came across this channel looking for a way to add GPU usage and tracing in our custom parabricks workflow, and wondering whether has been any new developments since the hackathon? I've had a peek at a few links - is the best repo to keep an eye on? https://github.com/FloWuenne/nextflow/tree/gpu-metrics-record

I'd be interested to help with testing if a new (parabricks) use case is required! useful 😋

do we have some examples with cuda setup and seqera containers? https://github.com/nf-core/modules/pull/10306/changes#r2884744009

Do we have general GPU testing support in nf-core/modules already? I am looking at it now but don't see accelerator set anywhere globally. Is our best practice to make a separate gpu nf-test and tag it so runners can pick it up?

I'm planning to create a rapids-singlecell module to GPU accelerate a few pipelines that use scverse tools like scanpy. However, there isn't a single scanpy module but rather a few that each provide individual steps like PCA, ... . I'd like to avoid having to create a module for the 50+ functions that RSC provides. I'm already astonished that this is best practice for scanpy because scanpy offers so much more than filtering, hashsolo, scrublet, and PCA. I know that other tools like neighbors, Leiden etc are used in various pipelines. Is a "master" rapids-singlecell module in scope or how should I go about it? @Nico Trummer Sorry, I'm a Nextflow boomer that last really implemented stuff in 2019 or so when modules were just about to become a thing.

Hi <!here>, We are facing an issue with GPU availability when running workflows via Seqera platform using awsbatch, even though the same setup works correctly when executed manually on a GPU instance. This issue is specifically observed in #CHN5BV5DW CellBender background removal step, which uses the process_gpu configuration. What We Tried 1. Instance Identification ◦ compute using the ▪︎ AMI: Amazon Linux AMI 2023.0.20260424 x86_64 ECS HVM EBS ▪︎ Instance type: g4 (GPU-enabled instance) 2. Manual GPU Instance Testing ◦ Using the same AMI, we launched a GPU instance manually. ◦ Verified execution locally: ▪︎ Ran the container interactively using Docker → ❌ Failed (CUDA not detected) ▪︎ Observed the same CUDA issue with the official nf-core GPU image. 3. Nextflow Local Execution ◦ Attempted to run via Nextflow → ❌ Same issue (

CUDA is not available

). 4. Fix Applied ◦ Identified that GPU access required explicit configuration. ◦ Added: --gpus all via

containerOptions

in Nextflow config. ◦ ✅ This resolved the issue: • Docker (interactive) → Working • Nextflow (local) → Working 5. Seqera Platform Execution ◦ Applied the same configuration in Seqera. ◦ ❌ Issue persists: GPU is still not detected (

CUDA not available

). 6. Additional Checks ◦ Verified accelerator operator → Already present ◦ Tried multiple configurations like containeroptions, docker.runoptions for that particular process → No success ❗ Current Issue Even after: • Using the same AMI, • Ensuring GPU instance type, • Passing

--gpus all

, • Confirming local success with Docker and Nextflow, GPU is not accessible when the pipeline is triggered from Seqera. Question: What could be causing GPU not being exposed/accessible inside containers when launched via Seqera?

What's the best practice for GPU modules? I saw nf-co.re/docs/developing/components/gpu-modules , but I'm wondering who and where

process.accelerator

(docs.seqera.io/nextflow/reference/process#…) should be set by. Does every module that checks

task.accelerator

have to have the

process_gpu

label? Not all currently do ( which means they default to cpu - why not default to gpu? ) Should pipelines need to set

process.accelerator

? The

-profile gpu

sets

task.accelerator = 1

but I guess this should be overridden by an actual map that assigns

type

and

request

. It would make sense that institutional configs set this - but if they support multiple types, how exactly should this be supplied by the user? Is a user expected to override settings with a custom config with process selectors and set the accelerator?

process {
    withName: 'GPU_PROCESS_ONE' {
        accelerator = [ type: 'h100', request: 4 ]
    }
    withName: 'GPU_PROCESS_TWO' {
        accelerator = [ type: 'l40s', request: 1 ]
    }
}

and then if they're using an executor that doesn't directly support the directive, should the institute config add it to say

process.clusterOptions

via the cluster directives e.g. slurm -> grus Is it turned off for a particular module in the same way

process {
    withName: 'GPU_PROCESS_ONE' {
        accelerator = null
    }
}

and if so, what's the use of

ext.use_gpu

now? And if overriding generally, I guess they should first override

withLabel: 'process_gpu'

and then use

withName

?