Hi spatialvi users, just curious if any of you can run successful the pipeline using singularity? I ran a simplest command but still get error message (discussed with Erik above).

nextflow run nf-core/spatialvi -profile singularity,test -r dev

Thank you.

I'm sorry to hear that you're still having issues. I assume you're still getting errors in the downstream parts of the pipeline? I can indeed run the pipeline using singularity. Do you have the option of trying with Docker or Apptainer instead, just to check that it's not some issue with your file permissions or similar?

It is still challenging using a tool with a very active author maintaining. For many bioinformatics tools without author's support, it is a nightmare of reproducibility for me.

Yes, I can understand that. It is important to remember that the spatialvi pipeline is still not released, and thus bugs and issues can be more common.

Hi Erik, I'm a staff member in the Research Computing group and I've been testing the pipeline on our cluster. Thank you for your continued help with the troubleshooting. Our cluster supports running Singularity but not Docker/Apptainer, so this is one constraint. I took a look at the error message and ran commands using the container. I think the issue is that the container (

docker.io-erikfas-spatialvi.img

) is not finding the file:

quality_controls.qmd

. I ran the following commands:

#Open a Shell Inside the Container:
scc-singularity shell /projectnb/ar-rcs/client/chdoan/work/singularity/docker.io-erikfas-spatialvi.img
Singularity> cd /projectnb/ar-rcs/client/chdoan/work/06/b663ebea5cbb6ce6e7fa86f5af00fb/
Singularity> quarto render \
>     quality_controls.qmd \
>     --execute-params params.yml \
>     --metadata subtitle:'Sample: CytAssist_11mm_FFPE_Human_Glioblastoma_2' \
>     --output quality_controls.html
# Starting python3 kernel...Done
# Executing 'quality_controls.ipynb'
#   Cell 1/17...Done
#   Cell 2/17...Done
#   Cell 3/17...Done
...
#   Cell 17/17...Done
# Output created: quality_controls.html

Note: I had to

cd

to the work folder and then run the

quarto

command. Perhaps there's an update needed for the Nextflow code? I'm also available to schedule a Zoom meeting with you if that would be easier for us to troubleshoot.

Very strange. That file is an input to the process that requires it, so I have no clue as to how it can't be present inside the container. The log files that were provided before didn't indicate that it was this file that was missing, but that is was Quarto that was missing some file when trying to run the report (possibly due to either permissions or the setup of the executing environment). Could you also provide the logfile for a failing run, @Andy Rampersaud?

Hi @Erik Fasterius, please see the log file attached. Error message happening around line 240.

What files are available inside the task directory after failure?

Hi @Erik Fasterius, I had to re-run the test but the error message is the same. Here's the contents of the work directory:

# Work dir:
#   /projectnb/ar-rcs/client/chdoan/work/59/cd829d8fff20da853552709b280d75
cd /projectnb/ar-rcs/client/chdoan/work/59/cd829d8fff20da853552709b280d75
ls -alht

-rw-r--r-- 1 aramp10 scv    1 Dec 18 11:41 .exitcode
drwxr-sr-x 4 aramp10 scv 4.0K Dec 18 11:41 .
-rw-r--r-- 1 aramp10 scv  534 Dec 18 11:41 .command.err
-rw-r--r-- 1 aramp10 scv  534 Dec 18 11:41 .command.log
drwxr-sr-x 4 aramp10 scv 4.0K Dec 18 11:41 .xdg_cache_home
drwxr-sr-x 2 aramp10 scv 4.0K Dec 18 11:41 artifacts
-rw-r--r-- 1 aramp10 scv  327 Dec 18 11:41 params.yml
-rw-r--r-- 1 aramp10 scv    0 Dec 18 11:41 .command.trace
-rw-r--r-- 1 aramp10 scv    0 Dec 18 11:41 .command.out
lrwxrwxrwx 1 aramp10 scv   76 Dec 18 11:41 _extensions -> /usr3/graduate/aramp10/.nextflow/assets/nf-core/spatialvi/assets/_extensions
lrwxrwxrwx 1 aramp10 scv   85 Dec 18 11:41 sdata_raw.zarr -> /projectnb/ar-rcs/client/chdoan/work/b2/83a9a7cf0a3e80d7e0b77620721dd8/sdata_raw.zarr
lrwxrwxrwx 1 aramp10 scv   82 Dec 18 11:41 quality_controls.qmd -> /usr3/graduate/aramp10/.nextflow/assets/nf-core/spatialvi/bin/quality_controls.qmd
-rw-r--r-- 1 aramp10 scv    0 Dec 18 11:41 .command.begin
-rw-r--r-- 1 aramp10 scv 1.6K Dec 18 11:41 .command.sh
-rw-r--r-- 1 aramp10 scv  11K Dec 18 11:41 .command.run

Okay, everything does seem to be there. Are you able to test with Docker or Apptainer and see if that works?

Hi @Erik Fasterius, yes, we use the same system.

Hi @Erik Fasterius, a fellow team member and I spent ~3 hours debugging the issue and found a working solution. We had noticed this in the .command.run file:

singularity exec --no-home --pid -B /usr3/graduate/aramp10/.nextflow/assets/nf-core/spatialvi -B /projectnb/ar-rcs/client/chdoan/work /projectnb/ar-rcs/client/chdoan/work/singularity/docker.io-erikfas-spatialvi.img

Note: the "work" folder is mounted. We tried many different things but these edits allowed it run: Edit this file: /usr3/graduate/aramp10/.nextflow/assets/nf-core/spatialvi/modules/nf-core/quartonotebook/main.nf

export TMPDIR=/projectnb/ar-rcs/client/chdoan/work

This export command would go above the

# Render notebook

command and then the test commands for the pipeline will run successfully. This is our current working solution and this will only work if the pipeline is run from this specific folder. Could you suggest a way to make this solution more general?

Nice debugging! That seems in line with what we discussed before, that the TMPDIR used by Singularity by default (

/tmp

) doesn't exist on your system. By looking a bit at nextflow and nf-core issues, I found one possible fix (here: https://github.com/nextflow-io/nextflow/issues/2685#issuecomment-1058238754) for you, by mounting /tmp to your $TMPDIR in Singularity: In the

nextflow.config

, add the following runOption value in the singularity block:

singularity {
        singularity.enabled     = true
        singularity.autoMounts  = true
        conda.enabled           = false
        docker.enabled          = false
        podman.enabled          = false
        shifter.enabled         = false
        charliecloud.enabled    = false
        apptainer.enabled       = false
        runOptions = "--bind \${TMPDIR}:/tmp"
    }

(Make sure you do have a $TMPDIR set, and that it exists) I don't know how I didn't see that when we discussed this earlier!

Hi @Christophe Avenel, thanks for the suggestion. I ran the test with the edit in the

nextflow.config

but unfortunately it failed with the same error (Quarto) message. When users submit jobs, our system does set TMPDIR:

echo $TMPDIR 
# /scratch/1656769.1.p-int

Another note: Regarding edits to the

.../quartonotebook/main.nf

, I noticed that this works:

export TMPDIR=/projectnb/ar-rcs/client/chdoan/work

But this will fail:

export TMPDIR=/projectnb/ar-rcs/client/chdoan

This leads me to think that there's something specific about Quarto that needs the "work" subfolder. Maybe there's something about not having permission/access to the parent folder. For now we have a working solution, perhaps someone can find a more general one that can work with our system.

Glad you found a work-around! The

/work

directory is just the name of the subdirectory inside the container that is defined in the image, there is nothing special about it per se and it could be named something different (so not Quarto-related). I don't know why your system has problems with this, but at least you got it working - I will close the related GitHub issue for now. It definitely seems that it's a system-specific problem here, so I think you'll have to talk to your sysadmin if you really want to get to the bottom of it.

Okay, thank you @Erik Fasterius for the extended support!