<@UE4L64T4G> Missed this in the release notes: <ht...
# tools
e
👌 1
party this 1
😵 1
m
or @Matthias Zepper
or actually @Júlia Mir since she did the #tools releases
m
Does that mean we don’t need https://github.com/nf-core/modules/pull/7260?
It appears there are additional updates in the PR, such as modifications to the conda’s
environment.yml
, which could still be relevant. However, there is no longer a strict requirement to replace
oras://
container declarations in modules, as Tools 3.1 now provides* basic* support for them. That said, I would greatly appreciate continued efforts towards the initial plan for properly adopting Seqera Containers, namely a dedicated
container.yml
/ container section in
meta.yml
within the pipeline repository. That file could be parsed by
nf-core pipelines download
for architecture-specific downloads. Currently, nf-core download cannot consolidate native Singularity and Docker images. As a result, in addition to the native Singularity images, it also downloads all Docker images of Seqera Containers and converts them to Singularity images. This means that for an average pipeline, you will download ~25 superfluous container images. From the Download perspective, Bioconda/Biocontainer images thus remain the preferred option over Seqera Containers, but if you choose them for your modules, the
oras://
protocol is now supported as well.
party this 3
p
@Edmund I think that https is still better, these arguments are all still relevant: https://nf-co.re/blog/2024/seqera-containers-part-2#singularity-oras-or-https
However, there are some downsides:
• Shared cache Nextflow options such as
$NXF_SINGULARITY_CACHEDIR
and
$NXF_SINGULARITY_LIBRARYDIR
are not used
• Singularity must be installed when downloading images for offline use
•
oras://
is only supported by recent versions of Singularity / Apptainer
As @Matthias Zepper says, we really just need to get on and implement the new Seqera Containers plan
quick look 2
💯 1
m
Ok, so we continue with https for now
And we'll deal with Oras when the time is right
e
Will download work if you have a really old singularity version that doesn't support oras?
m
No, because all that download does is pasting together and executing the Singularity command. But if the tool doesn’t support it, download does nothing by itself.
👍🏻 1
e
Hmm I wonder if we can check for the version that started supporting it, or just check for the failure and throw a helpful error "Your singularity is too old"
m
I have never tried it out, but I guess Singularity will throw an error in that case? If that happens, the error message will be shown to the user also in the context of nf-core download, so I do not see a reason to implement a specific error in nf-core for that.
l
I take this as "pro https" (at least for now) and will merge the PR, thanks for clarifying 😄
👍 2
👍🏻 1
m
Yeah, go for it
p
@Edmund is there any reason not to go for https though? We don't have private containers in nf-core and pulling https doesn't involve any build, so there's no heavy cpu work for the head node. Those are the only two pro-oras things I could think of in the above blog post.
So that leaves me with 3 pro-https things (cachedir / library dir, no need to have singularity installed for download, compatible with more versions of singularity) and 0 pro-oras things.
m
oras don't work with cachedir for now or it won't ever?
p
I don't think it ever well - the pull happens on the compute node with oras, and typically compute tasks don't have access to shared storage paths in the same way.. I think?
Felt increasingly unsure about that the more that I typed 😅
m
for me what I like with https links is that you don't need singularity
💯 1
p
But anyway, the plan is to support both https and oras - see the above blog post
Via
-profile singluarity
or
-profile singularity_oras
So folks will be able to choose
m
yeah then we need the
container.yml
and deal with it nicely
ok
I'm on board
p
This was the main blocker for moving forward by the way, and it's now merged: https://github.com/nextflow-io/nextflow/pull/5580
m
which release?
p
Not released yet, we did discuss back-porting to 24.10.x though
👍 1