Jin Khoo
04/14/2025, 8:45 AMdev to master but both "download (pull request)" and "download (pull_request_target)" failed. This is the error:
FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: opt/custflow/epi2meuser/dorado/lib/libdorado_torch_lib.so: unpack to regular file: short write: write /tmp/build-temp-327496813/rootfs/opt/custflow/epi2meuser/dorado/lib/libdorado_torch_lib.so: no space left on device
The failed tests:
https://github.com/nf-core/methylong/pull/6
seems like the failing is due to out of storage, any idea how to solve this? Many thanks!Jin Khoo
04/14/2025, 12:41 PMdownload test . Here I'm having the issue of running out of disk space, because dorado image is pretty large. Is there a way to increase disk space? or maybe skipping this module in the test?Matthias Zepper
04/14/2025, 2:17 PMjlumbroso/free-disk-space with this GitHub action step if it frees some additional GBs that suffice to make it work.
2.) You delete that workflow from the repo (effectively disabling the test) and perform a similar test manually prior to release. Personally, I would allow that, since the ultimate goal - that the pipeline is downloadable for offline use -is served.
That being said: The fact that it happens during the pipeline run stage of the test and not during the prior download step already tells me that the pipeline does not work with download as it is. It tries pulling an additional image at run time:
ERROR ~ Error executing process > 'NFCORE_METHYLONG:METHYLONG:ONT:ONT_ALIGN:DORADO_ALIGNER (2)'
Caused by:
Failed to pull singularity image
command: singularity pull --name registry.hub.docker.com-ontresearch-dorado-mr597_sha6058abbabae30a845dcc4ac7b481208de9d2af71.img.pulling.1744632422901 docker://registry.hub.docker.com/ontresearch/dorado:mr597_sha6058abbabae30a845dcc4ac7b481208de9d2af71 > /dev/null
status : 255
hint : Try and increase singularity.pullTimeout in the config (current is "20m")
message:
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
If you check the output it also tells you that downloading this particular image failed. Can you please manually try downloading the pipeline locally and explore that the reason for this is?Jin Khoo
04/15/2025, 7:43 AMnf-core pipelines download nf-core/methylong --revision dev --outdir ./methylong --compress "none" --container-system 'singularity' --container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io" --container-cache-utilisation 'amend' --download-configuration 'yes'
And the log
nf-core/tools version 3.2.0 - https://nf-co.re
WARNING Could not find GitHub authentication token. Some API requests may fail.
Nextflow and nf-core can use an environment variable called $NXF_SINGULARITY_CACHEDIR that is a path to a directory where remote Singularity images are stored. This allows downloaded images to be
cached in a central location.
? Define $NXF_SINGULARITY_CACHEDIR for a shared Singularity image download folder? [y/n]: y
? Specify the path: .
So that $NXF_SINGULARITY_CACHEDIR is always defined, you can add it to your ~/.bashrc file .This will then be automatically set every time you open a new terminal. We can add the following line to
this file for you:
export NXF_SINGULARITY_CACHEDIR="."
? Add to ~/.bashrc ? [y/n]: n
INFO Saving 'nf-core/methylong'
Pipeline revision: 'dev'
Use containers: 'singularity'
Container library: 'quay.io, docker.io, community.wave.seqera.io'
Using $NXF_SINGULARITY_CACHEDIR': /dss/dsslegfs01/pn73so/pn73so-dss-0000/becker_common/Jin/offline_methylong'
Output directory: './methylong'
Include default institutional configuration: 'True'
INFO Downloading centralised configs from GitHub
INFO Downloading workflow files from GitHub
INFO Processing workflow revision dev, found 13 container images in total.
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62% • 8/13 completed
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62% • 8/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 comple
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 c
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 13/13 completed
Not sure whats the cause of the different results here, but I switched to a different link to the image, now it works :)) thanks!Matthias Zepper
04/15/2025, 9:16 AMTry and increase singularity.pullTimeout is on the point?
I am not aware that I am explicitly setting and enlarging the time out somewhere for the test, but it could be set different on your local system? Alternatively, you were testing from a system with a better bandwidth and therefore you managed to pull the image without hitting a time out?