https://nf-co.re logo
Join Slack
Powered by
Hi all! I've been following the planned changes fo...
# tools
a
Hi all! I've been following the planned changes for the next release of nf-core tools, and I noticed some major updates to the pipeline download command, which is really nice! I was wondering, since it now relies on 
nextflow inspect
rather than regex patterns, would it be possible to allow users to pass a 
params.yml
file containing some parameters that could enable the selection of specific containers? The use case I have in mind is for GPU/CPU containers. We currently have both in our neuroimaging module library, which we can select at runtime, depending on whether we have a GPU or not. It would be cool if we could pass 
--gpu true
within a 
params.yml
file to the download command so it does not always download the CPU version. Thanks all! (PS. Not sure if my question is clear, happy to provide further details if needed!)
p
Is it a single process that flips the 
container
attribute? (vs. The param choosing between different processes?)
We are working our way towards having profiles with different containers for cpu architectures already. I wonder if we could generalise this somehow to also accept gpu (and other multiple accelerator types) 🤔
'cc <!subteam^S08LAB76NBU>
a
What we were moving towards was something like this:
Copy code
container "${ task.ext.use_gpu ? '<http://us.gcr.io/broad-dsde-methods/cellbender:0.3.2|us.gcr.io/broad-dsde-methods/cellbender:0.3.2>' :
        workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        '<https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/eb/ebcf140f995f79fcad5c17783622e000550ff6f171771f9fc4233484ee6f63cf/data>':
        '<http://community.wave.seqera.io/library/cellbender_webcolors:156d413fdfc16cdb|community.wave.seqera.io/library/cellbender_webcolors:156d413fdfc16cdb>' }"
This was taken from the cellbender/removebackground module.
So the choice between GPU or CPU would be given through the 
$task.ext
params to each module within a pipeline. However, I assume a pipeline would have a global 
--gpu
parameter that would toggle on/off this 
$task.ext
in each module, which is the one I would like to be able to give to the 
nf-core pipelines download
command.
p
ughh I want to get rid of all of that logic 🫠
😆 2
We are moving towards pipelines having different config profiles for cpu (in this case, gpu) which import config files that are purely list of containers: https://nf-co.re/blog/2024/seqera-containers-part-2#pipelines
Then auto-generate these files based on the contents of 
meta.yml
files for modules
So here the 
meta.yml
would look something like this:
Untitled.yaml
Untitled.yaml
Then you'd run with 
nextflow run <pipeline> -profile docker,gpu
config/containers_docker_gpu.config
would probably be quite short, only the modules with gpu defined. Then it'd overwrite the containers specified by 
config/containers_docker_amd64.config
for those processes.
What's your timeline? This is something we're actively trying to roll out. Could potentially use your pipeline as a guinea pig if you like, if you're not in a rush (it might take a while for the tooling to catch up with this)
a
I don't have a deadline or specific timeline for now. We recently released our containers with GPU/CPU specifications a few weeks ago and were seeking the best way to incorporate them into our neuroimaging modules (https://github.com/scilus/nf-neuro). My pipeline does not currently support GPU/CPU (https://github.com/scilus/nf-pediatric), but I'm happy to upgrade to our latest module versions if you'd like to test it out. However I like your suggestion, I'll do some reading on the blog posts you linked!
I'm assuming the autogeneration of these files (the ones in the blog post) has to be done by the maintainer at each module update, right?
Open in Slack
PreviousNext
Powered by