I m trying to download sarek v3 6 just released with nf core nf-core #tools

I’m trying to download sarek v3.6 (just released) ...

Anders Sune Pedersen

10/15/2025, 3:12 PM

I’m trying to download sarek v3.6 (just released) with nf-core/tools v3.3.2. I’m setting it up to download the singularity containers, but I get:

Copy code

ERROR    HTTPSConnectionPool(host='<http://depot.galaxyproject.org|depot.galaxyproject.org>', port=443): Max retries exceeded with
         url: /singularity/cnvkit:0.9.10--pyhdfd78af_0 (Caused by SSLError(SSLEOFError(8, '[SSL:
         UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1077)')))

I initially thought that depot.galaxyproject.org might be blocked on by the firewall on that machine, but that doesn’t seem to be the case, in fact, I can wget the cnvkit-container from the depot:

Copy code

wget <http://depot.galaxyproject.org/singularity/cnvkit:0.9.10--pyhdfd78af_0|depot.galaxyproject.org/singularity/cnvkit:0.9.10--pyhdfd78af_0>

😱 1

Anders Sune Pedersen

10/15/2025, 3:18 PM

The folder

nf-core-sarek_3.6.0/singularity-images

contains a bunch of symlinks like community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-92-927ff9bb80d65b425cbe752db6648a84043feff6e8ca90e60f9ff6ddbe8938d5-data.img -> ./blobs-sha256-92-927ff9bb80d65b425cbe752db6648a84043feff6e8ca90e60f9ff6ddbe8938d5-data.img and proper

.img

-files like:

Copy code

blobs-sha256-fc-fc33d59c090cef123aca26ae17fbddbd596640304d8325cbd5816229fa2c05ee-data.img

but no files with the names that I would expect.

mashehu

10/15/2025, 3:24 PM

can you try with nf-core/tools 3.4.0?

Anders Sune Pedersen

10/15/2025, 3:25 PM

Perhaps not so easy for me. I didn’t find v3.4 on bioconda. Is there any other conda-channel where I might find v3.4?

mashehu

10/15/2025, 3:26 PM

can you try

pip install --upgrade --force-reinstall git+<https://github.com/nf-core/tools.git@main>

? it was just released, so it always takes a bit to come up in conda channels

Slackbot

10/15/2025, 3:26 PM

Set up Bioconda channels:

Copy code

conda config --add channels bioconda
conda config --add channels conda-forge
conda config --set channel_priority strict

See https://bioconda.github.io/#usage

Anders Sune Pedersen

10/15/2025, 3:28 PM

Thanks for the command. I’ll give that a try. By the way, I’m also doing the download with nf-core v3.3.2 on another “normal” linux-box (instead of our HPC which is “crippled” by security measures), and it seems to run fine.

Anders Sune Pedersen

10/15/2025, 6:07 PM

Looks much more promising with nf-core/tools v3.4.0 🙂

🙌🏻 1

Anders Sune Pedersen

10/15/2025, 6:26 PM

Damn! Spoke to soon:

Copy code

Processing container images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% • 0/52 tasks completed
Pulling singularity images ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   4% • 2/52 tasks completed
<http://nvcr.io/nvidia/clara/clara-parabricks:4.5.1-1|nvcr.io/nvidia/clara/clara-parabricks:4.5.1-1> INFO:    Fetching OCI image... ━━━━━━━━━━━━━━━━━━━━ERROR    "The pipeline requested the download of non-existing container image
Processing container images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% • 0/52 tasks completed
Pulling singularity images ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   4% • 2/52 tasks completed
<http://nvcr.io/nvidia/clara/|nvcr.io/nvidia/clara/>… FATAL:   While making image from oci registry: error fetching image to c… INFO     Please try to rerun
         "singularity pull --name
         /ngc/people/appsadmin/fetch_sarek/nf-core-sarek_3.6.0/singularity-images/tmphl7gk0vd/tem
         pfile <docker://nvcr.io/nvidia/clara/clara-parabricks:4.5.1-1>" manually with a different
         registry.f

ERROR    "The pipeline requested the download of non-existing container image
         "<docker://nvcr.io/nvidia/clara/clara-parabricks:4.5.1-1>"

ERROR    Please try to rerun
         "singularity pull --name
         /ngc/people/appsadmin/fetch_sarek/nf-core-sarek_3.6.0/singularity-images/tmphl7gk0vd/tem
         pfile <docker://nvcr.io/nvidia/clara/clara-parabricks:4.5.1-1>" manually with a different
Processing container images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━  94% • 49/52 tasks completed
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 52/52 tasks completed
ERROR    ('Download errors',
         [SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://depot.galaxyproject.org|depot.galaxyproject.org>', port=443):
         Max retries exceeded with url: /singularity/ngscheckmate:1.0.1--py312pl5321h577a1d6_4
         (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in
         violation of protocol (_ssl.c:1077)')))")),
         SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://depot.galaxyproject.org|depot.galaxyproject.org>', port=443):
         Max retries exceeded with url: /singularity/samtools:1.20--h50ea8bc_0 (Caused by
         SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation
         of protocol (_ssl.c:1077)')))")),
...

Anders Sune Pedersen

10/15/2025, 6:29 PM

(I did manage to run

singularity pull <docker://nvcr.io/nvidia/clara/clara-parabricks:4.5.1-1>

on another linux-box 🤔 )

Anders Sune Pedersen

10/15/2025, 6:34 PM

On our HPC, I get

Copy code

$ singularity pull --name /ngc/people/appsadmin/fetch_sarek/nf-core-sarek_3.6.0/singularity-images/tmphl7gk0vd/tempfile <docker://nvcr.io/nvidia/clara/clara-parabricks:4.5.1-1>
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
INFO:    Fetching OCI image...
FATAL:   While making image from oci registry: error fetching image: while building SIF from layers: conveyor failed to get: while checking OCI image: Get "<https://layers.nvcr.io/registry/docker/registry/v2/blobs/sha256/3f/3f5818aa747f7f00136e8f03c40ec8dfc6677caa49dd63e1f6e3c516b906395b/data?Expires=REDACTED&Key-Pair-Id=REDACTED&Signature=REDACTED>": dial tcp 0.0.0.254:443: i/o timeout

I guess the problem is that

<http://layers.nvcr.io|layers.nvcr.io>

is blocked by the firewall:

Copy code

$ nslookup <http://layers.nvcr.io|layers.nvcr.io>
Server:         172.18.38.10
Address:        172.18.38.10#53

Non-authoritative answer:
Name:   <http://layers.nvcr.io|layers.nvcr.io>
Address: 0.0.0.254
;; Got SERVFAIL reply from 172.18.38.10, trying next server
** server can't find <http://layers.nvcr.io|layers.nvcr.io>: SERVFAIL

Matthias Zepper

10/16/2025, 8:56 AM

Regarding the first issue: Max retries exceeded could well have been a temporary issue when downloading multiple images from Galaxy via HTTP. We know for a while that all that is on depot.galaxyproject.org should according to Björn Grüning not be downloaded from there. Instead, we should use their distributed CVMFS, which is however poorly supported in Python ( the last time I checked there was one package unmaintained >5 years). Since the general direction is anyway to switch to Seqera Containers, we opted to ignore that, but it brings issues with the HTTP connection, such as failures and slow download speeds.

Matthias Zepper

10/16/2025, 8:57 AM

And regarding the symlinks: This was a feature request from you 😉

Anders Sune Pedersen

10/16/2025, 9:18 AM

Thanks for the additional explanation, Matthias. The symlinks are probably fine; just surprised to see the new naming scheme of containers used in sarek. Anyways, is there a way to run

nf-core pipelines download sarek

with a “resume” option?

Matthias Zepper

10/16/2025, 9:31 AM

No. But the pipeline download itself is quite tiny. What bogs you down are the Singularity images and if you have defined an

NXF_SINGULARITY_CACHEDIR

, they will not be re-downloaded, but copied from there, even if you have e.g. chosen

-u copy

-u remote

Anders Sune Pedersen

10/16/2025, 9:32 AM

Yes, without downloading the images, it is very fast. I didn’t have NFX_SINGULARITY_CACHEDIR defined. I’ll do that if the download fails again. Thanks.

Anders Sune Pedersen

10/16/2025, 11:58 AM

Unfortunately, it keeps failing:

Copy code

$ nf-core pipelines download sarek -r 3.6.0 --parallel-downloads 1 --container-system singularity


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.4.0 - <https://nf-co.re>

WARNING  Could not find GitHub authentication token. Some API requests may fail.
INFO     Detected Nextflow version 25.04.8

If transferring the downloaded files to another system, it can be convenient to have everything compressed in a
single file.
This is not recommended when downloading Singularity images, as it can take a long time and saves very little
space.
? Choose compression type: none
INFO     Saving 'nf-core/sarek'
         Pipeline revision: '3.6.0'
         Use containers: 'singularity'
         Container library: '<http://quay.io|quay.io>'
         Using NXF_SINGULARITY_CACHEDIR': /ngc/people/appsadmin/fetch_sarek/singularity-images'
         Output directory: 'nf-core-sarek_3.6.0'
         Include default institutional configuration: 'False'
INFO     Downloading workflow
INFO     Downloading workflow files from GitHub
INFO     Fetching container names for workflow revision 3.6.0 using nextflow inspect. This might take a while.
INFO     Processing workflow revision 3.6.0, found 52 container images in total.
Processing container images ━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  34% • 27/79 tasks completed
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 27/27 tasks completed
ERROR    ('Download errors', [SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://depot.galaxyproject.org|depot.galaxyproject.org>',
         port=443): Max retries exceeded with url: /singularity/cnvkit:0.9.10--pyhdfd78af_0 (Caused by
         SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol
         (_ssl.c:1077)')))")), SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://depot.galaxyproject.org|depot.galaxyproject.org>',
         port=443): Max retries exceeded with url:
         /singularity/mulled-v2-580d344d9d4a496cd403932da8765f9e0187774d:df80ed8d23d0a2c43181a2b3dd1b39f2d00fab5c-
         0 (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of
         protocol (_ssl.c:1077)')))")),
         SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://depot.galaxyproject.org|depot.galaxyproject.org>', port=443): Max retries
         exceeded with url:
         /singularity/mulled-v2-375a758a4ca8c128fb9d38047a68a9f4322d2acd:b3615e06ef17566f2988a215ce9e10808c1d08bf-
         0 (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of
         protocol (_ssl.c:1077)')))")),
         SSLError(MaxRetryError("HTTPSConnectionPool(host='<http://depot.galaxyproject.org|depot.galaxyproject.org>', port=443): Max retries …

Anders Sune Pedersen

10/16/2025, 12:28 PM

I’m down to fetching the containers like this:

Copy code

curl -L -o fastqc-0.12.1--hdfd78af_0.sif   <https://depot.galaxyproject.org/singularity/fastqc:0.12.1--hdfd78af_0>

🫤 1

oliviercoen

11/08/2025, 1:24 PM

or perhaps use Seqera containers?

2 Views

Open in Slack

Previous Next