nf-core/tools download does not support `oras` rig...
# team-maintainers
r
nf-core/tools download does not support
oras
right? We got quite a few modules still using it: https://github.com/search?q=org%3Anf-core%20oras%3A%2F%2F&type=code
m
That's correct as far as I'm aware. It depends on version if
oras
protocol is usable, and but the
https
version is usable by all I think. They should all be updated to
https
I believe.
party this 1
👍 1
r
@Júlia Mir is this something we can/should add linting for?
j
I think it's a good idea to lint for this until we can't support oras 👍🏻
m
I think Júlia needs an icon that triggers a "Make an issue for it 👍🏽" text.
🤣 4
😅 2
🙏 2
💯 2
s
I thought that @Matthias Zepper had made the tools support oras now: https://nfcore.slack.com/archives/CE5LG7WMB/p1739193058312809?thread_ts=1739177558.517239&cid=CE5LG7WMB
m
Doesn't look like it works all the time to use
oras
is how I read it.
s
That thread is talking about the fact you can't just stick
oras
on the front of the link to make it a singularity image 😉 But his comment is:
Copy code
the latest versions of nf-core pipelines download should actually support pulling oras://
👍🏽 1
p
One limitation of
oras
is that it requires having Singularity installed wherever you are running the command
I always quite liked that https downloads had no software requirements, but not sure how important that is to other people
m
Yes,
oras://
images are supported, so clearly no need to change the modules anymore. From the back of my head, I don't know if that code was already released, though or just merged to
dev
.
r
Great, thank you! 🙏 Missed that change
m
@Phil Ewels: Unfortunately, this makes no difference anymore. Since effectively all
oras://
URIs point to Seqera containers, one will still need Singularity in any case. (Because for all Seqera Container modules, it will download the Singularity image and convert the Docker image).
p
I don't follow?
oras://
from Seqera Containers will be Singularity images natively built with Singularity, that's what we do with Wave on the back end.
And if
https://
URLs are used for Seqera Containers, then the
.simg
files will be downloaded directly by Python without any external software requiement?
The other reason for not using
oras
more generally in nf-core is that it doesn't work with
$NXF_SINGULARITY_CACHEDIR
or
LIBRARYDIR
, which is probably not great news for a lot of HPC users
See discussion this part of the Seqera Containers - nf-core blog post
r
p
(Because for all Seqera Container modules, it will download the Singularity image and convert the Docker image).
What does this mean?
I think
https
is still better today, both because of how nextflow manages it (prefetching it from the monitoring process rather than having that happen in the job if needed, directories as mentioned). And having
https
allows e.g. using
nf-core
to fetch in a docker build (where I think there's not singularity being included today).
m
> (Because for all Seqera Container modules, it will download the Singularity image and convert the Docker image). > What does this mean? Exactly that. For any given pipeline, one will pull and unnecessarily convert a Docker image for each Seqera container. > And if
https://
URLs are used for Seqera Containers, then the
.simg
files will be downloaded directly by Python without any external software requierment? That is true. The
*.simg
files will be downloaded directly by Python without external software requirements. But the Docker images will be pulled as well that that is what requires the external Singularity/Apptainer installation. The reason for the unnecessary Docker pulls is the way how
nf-core pipelines download
is build. It gathers all container image paths from modules and config files in one big collection of findings and iteratively prunes them from duplicates and Docker equivalents for native Singularity images. This does no longer work for Seqera containers, as I have already pointed out on GitHub and in direct messages with you on SciLifeLab Slack. > The other reason for not using
oras
more generally in nf-core is that it doesn’t work with
$NXF_SINGULARITY_CACHEDIR
or
LIBRARYDIR
, which is probably not great news for a lot of HPC users > Shared cache Nextflow options such as
$NXF_SINGULARITY_CACHEDIR
and
$NXF_SINGULARITY_LIBRARYDIR
are not used. If your statement was unconditionally true, a workflow comprising a single
oras://
module could never be run offline?
nf-core pipelines download
at least creates a working offline image of an
oras://
version. You already pointed out the important difference that the oras:// pull will be executed from the process node instead of the head job, but if the
$NXF_SINGULARITY_CACHEDIR
/
NXF_APPTAINER_LIBRARYDIR
are accessible from the nodes as well, I hope this should still allow them to work with pre-downloaded versions?