Charles Plessy
05/31/2025, 6:49 AMnf-core pipelines download ignores it, but not the "Compare image container counts" step of the download CI in the pull request from dev to master, causing the test to fail. Maybe that is exactly the point, but I do not know. I can't decide between:
• report it as a bug in nf-core pipelines downlaod or the CI tests,
• workaround the problem by adding a stub local module that declares the container,
• revert the commit and ignore the redundancy between containers (except for mentioning how to consolidate in the pipeline's documentation).
What do you recommend?Charles Plessy
06/16/2025, 2:53 AMnf-core pipelines download that does not expect container override through conf/modules.config as I did it...Simon Pearce
06/16/2025, 4:21 AMnf-core modules patch to write a patch fileSimon Pearce
06/16/2025, 4:23 AMMatthias Zepper
06/16/2025, 7:56 AMnf-core download supports containers declared in configs, but relies on nextflow config to parse the configuration files. May I therefore ask if that container definition shows up when you run nextflow config --flat on your pipeline?Charles Plessy
06/16/2025, 10:02 PMprocess.'withName:ALIGNMENT_.*'.container = { "${workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'<https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/06/06beccfa4d48e5daf30dd8cee4f7e06fd51594963db0d5087ab695365b79903b/data>' :
'<http://community.wave.seqera.io/library/last_samtools_open-fonts:176a6ab0c8171057|community.wave.seqera.io/library/last_samtools_open-fonts:176a6ab0c8171057>'}" }Matthias Zepper
06/16/2025, 11:03 PMMatthias Zepper
06/18/2025, 4:30 PMconf/modules.config.
When I said that nf-core pipelines download supports container declared in configs, I was admittedly mostly thinking of single lines. They can contain a switching logic similar to yours (see the current test cases), but your config is already wrongly parsed by the utils function:
nf_core.utils.fetch_wf_config(Path(workflow_directory))
{'params.container': 'nfcore/methylseq:1.0', 'process.container': 'nfcore/methylseq:1.4', "process.'withName:Snpeff'.container": '{(params.annotation_cache && params.snpEff_cache) ? \'nfcore/sarek:dev\' : "nfcore/sareksnpeff:dev.${params.genome}"}', "process.'withName:Snpeff'.errorStrategy": "{task.exitStatus == 143 ? 'retry' : 'ignore'}", "process.'withLabel:VEP'.container": '{(params.annotation_cache && params.vep_cache) ? \'nfcore/sarek:dev\' : "nfcore/sarekvep:dev.${params.genome}"}', "process.'withLabel:VEP'.errorStrategy": "{task.exitStatus == 143 ? 'retry' : 'ignore'}", "process.'withName:RMARKDOWNNOTEBOOK'.conda": 'bioconda::r-shinyngs=1.7.1', "process.'withName:RMARKDOWNNOTEBOOK'.container": '{ "${ workflow.containerEngine == \'singularity\' && !task.ext.singularity_pull_docker_container ? \'<https://depot.galaxyproject.org/singularity/r-shinyngs:1.7.1--r42hdfd78af_1>\':\'quay.io/biocontainers/r-shinyngs:1.7.1--r42hdfd78af_1\' }" }', "process.'withName:ALIGNMENT_*.'.container": '{ "${workflow.containerEngine == \'singularity\' && !task.ext.singularity_pull_docker_container ?'}
while just above the other container string is fully extracted:
"process.'withName:RMARKDOWNNOTEBOOK'.container": '{ "${ workflow.containerEngine == \'singularity\' && !task.ext.singularity_pull_docker_container ? \'<https://depot.galaxyproject.org/singularity/r-shinyngs:1.7.1--r42hdfd78af_1>\':\'quay.io/biocontainers/r-shinyngs:1.7.1--r42hdfd78af_1\' }" }'
Yours is truncated:
"process.'withName:ALIGNMENT_*.'.container": '{ "${workflow.containerEngine == \'singularity\' && !task.ext.singularity_pull_docker_container ?'}
Therfore, nf-core pipelines download could not process it. In fact, the regex stumbled over it and extracted the singularity string instead, resulting in a mangled command and an attempted download of a Docker container of Singularity:
ERROR "singularity" failed for unclear reasons.
INFO Pulling of "singularity" failed.
Please troubleshoot the command
"singularity pull --name ./singularity_container_images/singularity.img <docker://community.wave.seqera.io/library//singularity>" manually.
Therefore, the container was missing.
Admittedly, I do not know yet why nf_core.utils.fetch_wf_config doesn’t catch it, but I recommend that for a short-term solution, you path the modules instead.
This is also the better option for nf-core pipelines download, because until we have been replacing the regexes with nextflow inspect, each Seqera container will be downloaded as Singularity image and converted from Docker. So contrary to the intuition, you do not save container download space by switching to the mulled container, but rather bloating your pipelines offline footprint 😬.