In a previous PR of <#C06TRU74N4D|>, I was told th...
# pipeline-maintainers
c
In a previous PR of #C06TRU74N4D, I was told that the contents of some of my containers overlap a lot, so I tried to consolidate them in pairgenomealign/commit/d824038…, reducing the container download mass by ~50%. Unfortunately,
nf-core pipelines download
ignores it, but not the "Compare image container counts" step of the
download
CI in the pull request from
dev
to
master
, causing the test to fail. Maybe that is exactly the point, but I do not know. I can't decide between: • report it as a bug in
nf-core pipelines downlaod
or the CI tests, • workaround the problem by adding a stub local module that declares the container, • revert the commit and ignore the redundancy between containers (except for mentioning how to consolidate in the pipeline's documentation). What do you recommend?
Hello! I would like to have some advice on an issue I encountered two weeks ago: • One PR reviewer suggested me to merge very overlapping containers for some of the modules of the #C06TRU74N4D pipeline and I did it, • Unfortunately this breaks
nf-core pipelines download
that does not expect container override through
conf/modules.config
as I did it...
s
Instead of overriding in the config, change the definitions in the module files and then use
nf-core modules patch
to write a patch file
(although it should probably also be reported as a bug)
👍 1
m
nf-core download
supports containers declared in configs, but relies on
nextflow config
to parse the configuration files. May I therefore ask if that container definition shows up when you run
nextflow config --flat
on your pipeline?
🙌 1
c
@Matthias Zepper yes it does…
Copy code
process.'withName:ALIGNMENT_.*'.container = { "${workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
            '<https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/06/06beccfa4d48e5daf30dd8cee4f7e06fd51594963db0d5087ab695365b79903b/data>' :
            '<http://community.wave.seqera.io/library/last_samtools_open-fonts:176a6ab0c8171057|community.wave.seqera.io/library/last_samtools_open-fonts:176a6ab0c8171057>'}" }
🤨 1
m
Ok, I will need to investigate that…
I took a look at the issue and the problem is indeed the mentioned container override in the
conf/modules.config
. When I said that
nf-core pipelines download
supports container declared in configs, I was admittedly mostly thinking of single lines. They can contain a switching logic similar to yours (see the current test cases), but your config is already wrongly parsed by the utils function:
Copy code
nf_core.utils.fetch_wf_config(Path(workflow_directory))
{'params.container': 'nfcore/methylseq:1.0', 'process.container': 'nfcore/methylseq:1.4', "process.'withName:Snpeff'.container": '{(params.annotation_cache && params.snpEff_cache) ? \'nfcore/sarek:dev\' : "nfcore/sareksnpeff:dev.${params.genome}"}', "process.'withName:Snpeff'.errorStrategy": "{task.exitStatus == 143 ? 'retry' : 'ignore'}", "process.'withLabel:VEP'.container": '{(params.annotation_cache && params.vep_cache) ? \'nfcore/sarek:dev\' : "nfcore/sarekvep:dev.${params.genome}"}', "process.'withLabel:VEP'.errorStrategy": "{task.exitStatus == 143 ? 'retry' : 'ignore'}", "process.'withName:RMARKDOWNNOTEBOOK'.conda": 'bioconda::r-shinyngs=1.7.1', "process.'withName:RMARKDOWNNOTEBOOK'.container": '{ "${ workflow.containerEngine == \'singularity\' && !task.ext.singularity_pull_docker_container ? \'<https://depot.galaxyproject.org/singularity/r-shinyngs:1.7.1--r42hdfd78af_1>\':\'quay.io/biocontainers/r-shinyngs:1.7.1--r42hdfd78af_1\' }" }', "process.'withName:ALIGNMENT_*.'.container": '{ "${workflow.containerEngine == \'singularity\' && !task.ext.singularity_pull_docker_container ?'}
while just above the other container string is fully extracted:
Copy code
"process.'withName:RMARKDOWNNOTEBOOK'.container": '{ "${ workflow.containerEngine == \'singularity\' && !task.ext.singularity_pull_docker_container ? \'<https://depot.galaxyproject.org/singularity/r-shinyngs:1.7.1--r42hdfd78af_1>\':\'quay.io/biocontainers/r-shinyngs:1.7.1--r42hdfd78af_1\' }" }'
Yours is truncated:
Copy code
"process.'withName:ALIGNMENT_*.'.container": '{ "${workflow.containerEngine == \'singularity\' && !task.ext.singularity_pull_docker_container ?'}
Therfore, nf-core pipelines download could not process it. In fact, the regex stumbled over it and extracted the
singularity
string instead, resulting in a mangled command and an attempted download of a Docker container of Singularity:
Copy code
ERROR  "singularity" failed for unclear reasons. 
INFO  Pulling of "singularity" failed. 
 Please troubleshoot the command 
 "singularity pull --name ./singularity_container_images/singularity.img  <docker://community.wave.seqera.io/library//singularity>" manually.
Therefore, the container was missing. Admittedly, I do not know yet why
nf_core.utils.fetch_wf_config
doesn’t catch it, but I recommend that for a short-term solution, you path the modules instead. This is also the better option for
nf-core pipelines download
, because until we have been replacing the regexes with
nextflow inspect
, each Seqera container will be downloaded as Singularity image and converted from Docker. So contrary to the intuition, you do not save container download space by switching to the mulled container, but rather bloating your pipelines offline footprint 😬.