https://nf-co.re logo
Join Slack
Powered by
Hi, Sharing <this> from <@U07FNBFRQCE> original po...
# modules
j
Hi, Sharing this from @Keiran Rowell original posted on #C08THK11CHX as I think it could be relevant for anyone working with GPU-enabled modules
j
I think that sort of depends
To be fair I've not worked on GPU modules yet so I know exactly how it works, but I wonder if it would be OK to say that 
--gpu
changes a lot of extra things outside the process itself (you need to dynamically set an 
accelerator
right? or have different tags), that maybe it makes sense to either have the 
gpu
mode as an inputs and set things accordingly Or maybe we allow hardcoded gpu-variant moduiles... So 
mmseqs/makepaddedseqdb
and 
mmseqs/makepaddedseqdb
where the latter has hardcoded 
--gpu
, the correct 
tag gpu
etc..? <!subteam^S044DU0NG68>, thoguhts?
e
Separate modules for GPU, and CPU-only usage makes sense to me
this 1
👍🏻 1
👍 1
I've done it in the past for my non nf-core pipelines
j
See also for https://nfcore.slack.com/archives/CQY2U5QU9/p1771579583230519 for anyone who hasn't joined the fun
a
An argument like 
--gpu
should be in ext.args though, so a pipeline developer can control it how they want. But I guess a 
val(gpu)
boolean would also work?
As James says there's a bunch of things you need to modify, like 
accelerator
, 
containerOptions
j
An argument like --gpu should be in ext.args though, so a pipeline developer can control it how they want.
To slightly derail the question for my own knowledge purposes:, what else do we actually need to modify to get a tool to work with GPU in a Nextflow module/pipeline? Even if a user could supply 
--gpu
, what else do we need to change within Nextflow to get it to work? Is it something that a pipeline developer could already account for, or does it require too much customisation by the user?
Becuase then we could make some guidance/specs/docs on how to make a GPU module, and how to implement it in a pipeline
e
-profile singularity,test,gpu
for example, add the profile there
n
I don't have much time right now, but here is the PR where we introduced the 
gpu
profile - I added a lot of explanations on the rational there
🤩 1
a
process.containerOptions = "--gpus $n" or "--nv" or others
process.accelerator = 1
process.machineType = "name-of-machine-with-gpu"
oh and 
process.queue = "queue-with-gpu"
k
Thanks so much everyone for providing the approaches and opinions! We'll read through them all as we add it into he config. The discussions in the linked PR are very helpful, reading through Nico's and Jonathan's attached discussions. We're in the lucky space that we can assure the 
'process_gpu'
label because its directly tied into the structure inference, so that narrows our solution space. Thanks again :)
2 Views
Open in Slack
PreviousNext
Powered by