Hello. I’m working on tools#3019 (Support NXF_SINGULARITY_LIBRARYDIR in

nf-core download

) and I’ve got a question about the container symlinks that are created, e.g.

The base image, e.g. ./nf-core-gatk-4.4.0.0.img will thus be symlinked as for example ./quay.io-nf-core-gatk-4.4.0.0.img

The function is nf_core/pipelines/download.py#L1030. I’ve got trouble understanding why: 1. If

NXF_SINGULARITY_CACHEDIR

is not set, symlinks are always created in the download directory after download (L1352 and L1463: in both cases,

output_path

is set to

out_path

because

cache_path

is not set) 2. If

NXF_SINGULARITY_CACHEDIR

is set, symlinks are created in the download directory only if the container is copied from the cache. Otherwise, symlinks are only created in the cache directory (still L1352 and L1463 but this time,

output_path

is set to

cache_path

). It feels inconsistent. Am I missing something ? Example with

nf-core pipelines download nf-core/fetchngs --revision 1.11.0 --download-configuration --compress none --container-system singularity --container-cache-utilisation copy

If

NXF_SINGULARITY_CACHEDIR

is set and complete, I get files + symlinks under

nf-core-fetchngs_1.11.0/singularity-images/

depot.galaxyproject.org-singularity-mulled-v2-5f89fe0cd045cb1d615630b9261a1d17943a9b6a-2f4a4c900edd6801ff0068c2b3048b4459d119eb-0.img -> ./singularity-mulled-v2-5f89fe0cd045cb1d615630b9261a1d17943a9b6a-2f4a4c900edd6801ff0068c2b3048b4459d119eb-0.img
depot.galaxyproject.org-singularity-python-3.9--1.img -> ./singularity-python-3.9--1.img
depot.galaxyproject.org-singularity-sra-tools-2.11.0--pl5321ha49a11a_3.img -> ./singularity-sra-tools-2.11.0--pl5321ha49a11a_3.img
depot.galaxyproject.org-singularity-sra-tools-3.0.8--h9f5acd7_0.img -> ./singularity-sra-tools-3.0.8--h9f5acd7_0.img
depot.galaxyproject.org-singularity-synapseclient-2.7.1--pyh7cba7a3_0.img -> ./singularity-synapseclient-2.7.1--pyh7cba7a3_0.img
depot.galaxyproject.org-singularity-ubuntu-20.04.img -> ./singularity-ubuntu-20.04.img
quay.io-singularity-mulled-v2-5f89fe0cd045cb1d615630b9261a1d17943a9b6a-2f4a4c900edd6801ff0068c2b3048b4459d119eb-0.img -> ./singularity-mulled-v2-5f89fe0cd045cb1d615630b9261a1d17943a9b6a-2f4a4c900edd6801ff0068c2b3048b4459d119eb-0.img
quay.io-singularity-python-3.9--1.img -> ./singularity-python-3.9--1.img
quay.io-singularity-sra-tools-2.11.0--pl5321ha49a11a_3.img -> ./singularity-sra-tools-2.11.0--pl5321ha49a11a_3.img
quay.io-singularity-sra-tools-3.0.8--h9f5acd7_0.img -> ./singularity-sra-tools-3.0.8--h9f5acd7_0.img
quay.io-singularity-synapseclient-2.7.1--pyh7cba7a3_0.img -> ./singularity-synapseclient-2.7.1--pyh7cba7a3_0.img
quay.io-singularity-ubuntu-20.04.img -> ./singularity-ubuntu-20.04.img

singularity-mulled-v2-5f89fe0cd045cb1d615630b9261a1d17943a9b6a-2f4a4c900edd6801ff0068c2b3048b4459d119eb-0.img
singularity-python-3.9--1.img
singularity-sra-tools-2.11.0--pl5321ha49a11a_3.img
singularity-sra-tools-3.0.8--h9f5acd7_0.img
singularity-synapseclient-2.7.1--pyh7cba7a3_0.img
singularity-ubuntu-20.04.img

On the other hand, If

NXF_SINGULARITY_CACHEDIR

is set but empty, the cache has the files and the symlinks as above, but I get only files under

nf-core-fetchngs_1.11.0/singularity-images/

$ ls nf-core-fetchngs_1.11.0/singularity-images/ 
singularity-mulled-v2-5f89fe0cd045cb1d615630b9261a1d17943a9b6a-2f4a4c900edd6801ff0068c2b3048b4459d119eb-0.img
singularity-python-3.9--1.img
singularity-sra-tools-2.11.0--pl5321ha49a11a_3.img
singularity-sra-tools-3.0.8--h9f5acd7_0.img
singularity-synapseclient-2.7.1--pyh7cba7a3_0.img
singularity-ubuntu-20.04.img

No, you are not missing something. That is indeed not the desired behaviour. I think, I just mixed up the two legacy variable names

out_path

and

output_path

when calling

self.symlink_singularity_images()

. Would be great if you could include a bugfix with your PR. Thanks!

Yep, I can do that, no pb. Thank you for your answer !

Good question 🤔. Because I only use the

-u remote

option normally, I did never really pay attention to what happened in

NXF_SINGULARITY_CACHEDIR

on the machine that I was downloading with - so I am flabbergasted that I never noticed the missing symlinks in the

singularity-images

folder, because there you need them for sure. Somebody who uses

-u amend

would need the symlinks the the cache also, so they should be there as well. The only downside to symlinking in the cache would be that there it is much more likely that a naming collision happens. Say if there is an Image with the same name but different contents in two different registries - then, the tool would falsely assume that it has the image already in cache and pull out the wrong one.

All right. I can make the code resilient to files already existing. I think the end result is that the

NXF_SINGULARITY_CACHEDIR

and output directories need to be complete, and the test is therefore: can nextflow run the pipeline without having to download anything, testing both directories in copy/amend modes etc. If that’s the case, then I think we’re good

That would be fantastic, but also sounds like a lot of work! (Take your time, I will be on holiday next week, so no PR reviews then 😬)