Rike Hanssen
03/31/2025, 8:15 AMoras right? We got quite a few modules still using it: https://github.com/search?q=org%3Anf-core%20oras%3A%2F%2F&type=codeMahesh Binzer-Panchal
03/31/2025, 8:33 AMoras protocol is usable, and but the https version is usable by all I think. They should all be updated to https I believe.Rike Hanssen
03/31/2025, 8:33 AMJúlia Mir
03/31/2025, 8:52 AMMahesh Binzer-Panchal
03/31/2025, 8:56 AMSimon Pearce
03/31/2025, 8:57 AMMahesh Binzer-Panchal
03/31/2025, 9:00 AMoras is how I read it.Simon Pearce
03/31/2025, 9:03 AMoras on the front of the link to make it a singularity image 😉
But his comment is:
the latest versions of nf-core pipelines download should actually support pulling oras://Phil Ewels
03/31/2025, 9:14 AMoras is that it requires having Singularity installed wherever you are running the commandPhil Ewels
03/31/2025, 9:14 AMMatthias Zepper
03/31/2025, 9:14 AMoras:// images are supported, so clearly no need to change the modules anymore. From the back of my head, I don't know if that code was already released, though or just merged to dev.Rike Hanssen
03/31/2025, 9:16 AMMatthias Zepper
03/31/2025, 9:18 AMoras:// URIs point to Seqera containers, one will still need Singularity in any case. (Because for all Seqera Container modules, it will download the Singularity image and convert the Docker image).Phil Ewels
03/31/2025, 9:19 AMoras:// from Seqera Containers will be Singularity images natively built with Singularity, that's what we do with Wave on the back end.Phil Ewels
03/31/2025, 9:19 AMhttps:// URLs are used for Seqera Containers, then the .simg files will be downloaded directly by Python without any external software requiement?Phil Ewels
03/31/2025, 9:21 AMoras more generally in nf-core is that it doesn't work with $NXF_SINGULARITY_CACHEDIR or LIBRARYDIR, which is probably not great news for a lot of HPC usersPhil Ewels
03/31/2025, 9:21 AMRike Hanssen
03/31/2025, 9:26 AMpontus
03/31/2025, 9:27 AM(Because for all Seqera Container modules, it will download the Singularity image and convert the Docker image).What does this mean?
pontus
03/31/2025, 9:34 AMhttps is still better today, both because of how nextflow manages it (prefetching it from the monitoring process rather than having that happen in the job if needed, directories as mentioned).
And having https allows e.g. using nf-core to fetch in a docker build (where I think there's not singularity being included today).Matthias Zepper
03/31/2025, 10:28 AMhttps:// URLs are used for Seqera Containers, then the .simg files will be downloaded directly by Python without any external software requierment?
That is true. The *.simg files will be downloaded directly by Python without external software requirements. But the Docker images will be pulled as well that that is what requires the external Singularity/Apptainer installation.
The reason for the unnecessary Docker pulls is the way how nf-core pipelines download is build. It gathers all container image paths from modules and config files in one big collection of findings and iteratively prunes them from duplicates and Docker equivalents for native Singularity images. This does no longer work for Seqera containers, as I have already pointed out on GitHub and in direct messages with you on SciLifeLab Slack.
> The other reason for not using oras more generally in nf-core is that it doesn’t work with $NXF_SINGULARITY_CACHEDIR or LIBRARYDIR, which is probably not great news for a lot of HPC users
> Shared cache Nextflow options such as $NXF_SINGULARITY_CACHEDIR and $NXF_SINGULARITY_LIBRARYDIR are not used.
If your statement was unconditionally true, a workflow comprising a single oras:// module could never be run offline?
nf-core pipelines download at least creates a working offline image of an oras:// version. You already pointed out the important difference that the oras:// pull will be executed from the process node instead of the head job, but if the $NXF_SINGULARITY_CACHEDIR / NXF_APPTAINER_LIBRARYDIR are accessible from the nodes as well, I hope this should still allow them to work with pre-downloaded versions?