Hi all, I made a PR to merge `dev` to `master` but...
# tools
j
Hi all, I made a PR to merge
dev
to
master
but both "download (pull request)" and "download (pull_request_target)" failed. This is the error:
Copy code
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: opt/custflow/epi2meuser/dorado/lib/libdorado_torch_lib.so: unpack to regular file: short write: write /tmp/build-temp-327496813/rootfs/opt/custflow/epi2meuser/dorado/lib/libdorado_torch_lib.so: no space left on device
The failed tests: https://github.com/nf-core/methylong/pull/6 seems like the failing is due to out of storage, any idea how to solve this? Many thanks!
Hi @Matthias Zepper, I noticed in an older thread u discussed about the issues with the
download test
. Here I'm having the issue of running out of disk space, because dorado image is pretty large. Is there a way to increase disk space? or maybe skipping this module in the test?
m
Unfortunately the test is not smart enough to allow declaring a module that should be skipped. The infrastructure team is in the process of moving certain tests from GitHub runners to RunsOn, but I am not aware that this has been implemented for this test and don’t know if that would actually mean bigger runners than before. Maybe @mashehu can shed some light on this? To fix your problem short term, you have two options. 1.) You could try replacing the free disk space step
jlumbroso/free-disk-space
with this GitHub action step if it frees some additional GBs that suffice to make it work. 2.) You delete that workflow from the repo (effectively disabling the test) and perform a similar test manually prior to release. Personally, I would allow that, since the ultimate goal - that the pipeline is downloadable for offline use -is served. That being said: The fact that it happens during the pipeline run stage of the test and not during the prior download step already tells me that the pipeline does not work with download as it is. It tries pulling an additional image at run time:
Copy code
ERROR ~ Error executing process > 'NFCORE_METHYLONG:METHYLONG:ONT:ONT_ALIGN:DORADO_ALIGNER (2)' 


Caused by: 

 Failed to pull singularity image 

 command: singularity pull --name registry.hub.docker.com-ontresearch-dorado-mr597_sha6058abbabae30a845dcc4ac7b481208de9d2af71.img.pulling.1744632422901 docker://registry.hub.docker.com/ontresearch/dorado:mr597_sha6058abbabae30a845dcc4ac7b481208de9d2af71 > /dev/null 

 status : 255 

 hint : Try and increase singularity.pullTimeout in the config (current is "20m") 

 message: 

 INFO: Converting OCI blobs to SIF format 


 INFO: Starting build...
If you check the output it also tells you that downloading this particular image failed. Can you please manually try downloading the pipeline locally and explore that the reason for this is?
j
hmm,, strange, I didnt notice it failed during the download step. Thanks for pointing it out. I tried downloading the pipeline locally with singularity and it worked fine. This was my command:
Copy code
nf-core pipelines download nf-core/methylong  --revision dev  --outdir ./methylong  --compress "none"  --container-system 'singularity'  --container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io" --container-cache-utilisation 'amend' --download-configuration 'yes'
And the log
Copy code
nf-core/tools version 3.2.0 - https://nf-co.re


WARNING  Could not find GitHub authentication token. Some API requests may fail.                                                                                                                        

Nextflow and nf-core can use an environment variable called $NXF_SINGULARITY_CACHEDIR that is a path to a directory where remote Singularity images are stored. This allows downloaded images to be 
cached in a central location.
? Define $NXF_SINGULARITY_CACHEDIR for a shared Singularity image download folder? [y/n]: y
? Specify the path: .

So that $NXF_SINGULARITY_CACHEDIR is always defined, you can add it to your ~/.bashrc file .This will then be automatically set every time you open a new terminal. We can add the following line to 
this file for you: 
export NXF_SINGULARITY_CACHEDIR="."
? Add to ~/.bashrc ? [y/n]: n
INFO     Saving 'nf-core/methylong'                                                                                                                                                                     
          Pipeline revision: 'dev'                                                                                                                                                                      
          Use containers: 'singularity'                                                                                                                                                                 
          Container library: 'quay.io, docker.io, community.wave.seqera.io'                                                                                                                             
          Using $NXF_SINGULARITY_CACHEDIR': /dss/dsslegfs01/pn73so/pn73so-dss-0000/becker_common/Jin/offline_methylong'                                                                                 
          Output directory: './methylong'                                                                                                                                                               
          Include default institutional configuration: 'True'                                                                                                                                           
INFO     Downloading centralised configs from GitHub                                                                                                                                                    
INFO     Downloading workflow files from GitHub                                                                                                                                                         
INFO     Processing workflow revision dev, found 13 container images in total.                                                                                                                          
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  62% • 8/13 completed
Downloading singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  62% • 8/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 comple
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 c
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━  77% • 10/13 completed
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 13/13 completed
Not sure whats the cause of the different results here, but I switched to a different link to the image, now it works :)) thanks!
m
Maybe the hint
Try and increase singularity.pullTimeout
is on the point? I am not aware that I am explicitly setting and enlarging the time out somewhere for the test, but it could be set different on your local system? Alternatively, you were testing from a system with a better bandwidth and therefore you managed to pull the image without hitting a time out?
✅ 1