Hi
I’m trying to test run a nf-core pipeline on m...
# helpm
Mark Pampuch
08/29/2024, 10:10 AMHi
I’m trying to test run a nf-core pipeline on my HPC but I keep running into errors regardless of what profile I use (docker, singularity, conda, etc).
See thread for code and error.
Mark Pampuch
08/29/2024, 10:11 AMHere is my sbatch file
Copy code
#!/bin/bash --login
# Set your job walltime (D-HH:MM:SS)
#SBATCH --time=1:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=100
#SBATCH --ntasks=1
#SBATCH --mem=32G
#SBATCH --partition=batch
#SBATCH --job-name=bacass
#SBATCH --mail-type=ALL
# Activate your conda environment for your job
conda activate "$(pwd)/env"
# Load singularity
module load singularity
# Run your job script here
nextflow run nf-core/bacass -r 2.1.0 -profile test,singularity --outdir results Copy code
N E X T F L O W ~ version 24.04.4
Launching `<https://github.com/nf-core/bacass>` [soggy_jennings] DSL2 - revision: e94a173311 [2.1.0]
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/bacass v2.1.0-ge94a173
------------------------------------------------------
Core Nextflow options
revision : 2.1.0
runName : soggy_jennings
containerEngine : singularity
launchDir : /ibex/user/pampum/2024-08-29_nextflow-test
workDir : /ibex/user/pampum/2024-08-29_nextflow-test/work
projectDir : /home/pampum/.nextflow/assets/nf-core/bacass
userName : pampum
profile : test,singularity
configFiles :
Input/output options
input : <https://raw.githubusercontent.com/nf-core/test-datasets/bacass/bacass_short.tsv>
outdir : results
Assembly parameters
unicycler_args : --no_correct --no_pilon
canu_mode : -nanopore
Annotation
prokka_args : --fast
dfast_config : /home/pampum/.nextflow/assets/nf-core/bacass/assets/test_config_dfast.py
Skipping Options
skip_kraken2 : true
skip_pycoqc : true
Institutional config options
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/bacass for your analysis please cite:
* The pipeline
<https://doi.org/10.5281/zenodo.2669428>
* The nf-core framework
<https://doi.org/10.1038/s41587-020-0439-x>
* Software dependencies
<https://github.com/nf-core/bacass/blob/master/CITATIONS.md>
[- ] NFC…IM_FASTP_FASTQC:FASTQC_RAW -
[- ] NFC…TQ_TRIM_FASTP_FASTQC:FASTP -
[- ] NFC…M_FASTP_FASTQC:FASTQC_TRIM -
[- ] NFCORE_BACASS:BACASS:NANOPLOT -
[- ] NFC…IM_FASTP_FASTQC:FASTQC_RAW -
[- ] NFC…TQ_TRIM_FASTP_FASTQC:FASTP -
[- ] NFC…M_FASTP_FASTQC:FASTQC_TRIM -
[- ] NFCORE_BACASS:BACASS:NANOPLOT -
[- ] NFCORE_BACASS:BACASS:UNICYCLER -
[- ] NFCORE_BACASS:BACASS:QUAST -
[- ] NFCORE_BACASS:BACASS:GUNZIP -
[- ] NFCORE_BACASS:BACASS:PROKKA -
[- ] NFC…USTOM_DUMPSOFTWAREVERSIONS -
[- ] NFC…IM_FASTP_FASTQC:FASTQC_RAW -
[- ] NFC…TQ_TRIM_FASTP_FASTQC:FASTP -
[- ] NFC…M_FASTP_FASTQC:FASTQC_TRIM -
[- ] NFCORE_BACASS:BACASS:NANOPLOT -
[- ] NFCORE_BACASS:BACASS:UNICYCLER -
[- ] NFCORE_BACASS:BACASS:QUAST -
[- ] NFCORE_BACASS:BACASS:GUNZIP -
[- ] NFCORE_BACASS:BACASS:PROKKA -
[- ] NFC…USTOM_DUMPSOFTWAREVERSIONS -
[- ] NFCORE_BACASS:BACASS:MULTIQC -
[- ] NFC…IM_FASTP_FASTQC:FASTQC_RAW -
[- ] NFC…TQ_TRIM_FASTP_FASTQC:FASTP -
[- ] NFC…M_FASTP_FASTQC:FASTQC_TRIM -
[- ] NFCORE_BACASS:BACASS:NANOPLOT -
[- ] NFCORE_BACASS:BACASS:UNICYCLER -
[- ] NFCORE_BACASS:BACASS:QUAST -
[- ] NFCORE_BACASS:BACASS:GUNZIP -
[- ] NFCORE_BACASS:BACASS:PROKKA -
[- ] NFC…USTOM_DUMPSOFTWAREVERSIONS -
[- ] NFCORE_BACASS:BACASS:MULTIQC -
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR044595_1M_1.fastq.gz>
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR064912_1M_1.fastq.gz>
[- ] NFC…IM_FASTP_FASTQC:FASTQC_RAW -
[- ] NFC…TQ_TRIM_FASTP_FASTQC:FASTP -
[- ] NFC…M_FASTP_FASTQC:FASTQC_TRIM -
[- ] NFCORE_BACASS:BACASS:NANOPLOT -
[- ] NFCORE_BACASS:BACASS:UNICYCLER -
[- ] NFCORE_BACASS:BACASS:QUAST -
[- ] NFCORE_BACASS:BACASS:GUNZIP -
[- ] NFCORE_BACASS:BACASS:PROKKA -
[- ] NFC…USTOM_DUMPSOFTWAREVERSIONS -
[- ] NFCORE_BACASS:BACASS:MULTIQC -
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR044595_1M_1.fastq.gz>
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR064912_1M_1.fastq.gz>
Pulling Singularity image <https://depot.galaxyproject.org/singularity/fastp:0.23.4--h5f740d0_0> [cache /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity/depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img]
Pulling Singularity image <https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0> [cache /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity/depot.galaxyproject.org-singularity-fastqc-0.11.9--0.img]
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity -- Use the environment variable NXF_SINGULARITY_CACHEDIR to specify a different location
[- ] NFC…IM_FASTP_FASTQC:FASTQC_RAW -
[- ] NFC…TQ_TRIM_FASTP_FASTQC:FASTP -
[- ] NFC…M_FASTP_FASTQC:FASTQC_TRIM -
[- ] NFCORE_BACASS:BACASS:NANOPLOT -
[- ] NFCORE_BACASS:BACASS:UNICYCLER -
[- ] NFCORE_BACASS:BACASS:QUAST -
[- ] NFCORE_BACASS:BACASS:GUNZIP -
[- ] NFCORE_BACASS:BACASS:PROKKA -
[- ] NFC…USTOM_DUMPSOFTWAREVERSIONS -
[- ] NFCORE_BACASS:BACASS:MULTIQC -
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR044595_1M_1.fastq.gz>
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR064912_1M_1.fastq.gz>
Pulling Singularity image <https://depot.galaxyproject.org/singularity/fastp:0.23.4--h5f740d0_0> [cache /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity/depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img]
Pulling Singularity image <https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0> [cache /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity/depot.galaxyproject.org-singularity-fastqc-0.11.9--0.img]
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity -- Use the environment variable NXF_SINGULARITY_CACHEDIR to specify a different location
ERROR ~ Error executing process > 'NFCORE_BACASS:BACASS:FASTQ_TRIM_FASTP_FASTQC:FASTQC_RAW (1)'
Caused by:
Failed to pull singularity image
command: singularity pull --name depot.galaxyproject.org-singularity-fastqc-0.11.9--0.img.pulling.1724926586610 <https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0> > /dev/null
status : 255
hint : Try and increase singularity.pullTimeout in the config (current is "20m")
message:
ERROR: pull is only supported for shub URIs
-- Check '.nextflow.log' file for details
-[nf-core/bacass] Pipeline completed with errors-
[- ] NFC…IM_FASTP_FASTQC:FASTQC_RAW -
[- ] NFC…TQ_TRIM_FASTP_FASTQC:FASTP -
[- ] NFC…M_FASTP_FASTQC:FASTQC_TRIM -
[- ] NFCORE_BACASS:BACASS:NANOPLOT -
[- ] NFCORE_BACASS:BACASS:UNICYCLER -
[- ] NFCORE_BACASS:BACASS:QUAST -
[- ] NFCORE_BACASS:BACASS:GUNZIP -
[- ] NFCORE_BACASS:BACASS:PROKKA -
[- ] NFC…USTOM_DUMPSOFTWAREVERSIONS -
[- ] NFCORE_BACASS:BACASS:MULTIQC -
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR044595_1M_1.fastq.gz>
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR064912_1M_1.fastq.gz>
Pulling Singularity image <https://depot.galaxyproject.org/singularity/fastp:0.23.4--h5f740d0_0> [cache /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity/depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img]
Pulling Singularity image <https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0> [cache /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity/depot.galaxyproject.org-singularity-fastqc-0.11.9--0.img]
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /ibex/user/pampum/2024-08-29_nextflow-test/work/singularity -- Use the environment variable NXF_SINGULARITY_CACHEDIR to specify a different location
ERROR ~ Error executing process > 'NFCORE_BACASS:BACASS:FASTQ_TRIM_FASTP_FASTQC:FASTQC_RAW (1)'
Caused by:
Failed to pull singularity image
command: singularity pull --name depot.galaxyproject.org-singularity-fastqc-0.11.9--0.img.pulling.1724926586610 <https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0> > /dev/null
status : 255
hint : Try and increase singularity.pullTimeout in the config (current is "20m")
message:
ERROR: pull is only supported for shub URIs
-- Check '.nextflow.log' file for details
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/bacass/ERR064912_1M_2.fastq.gz>p
pontus
08/29/2024, 10:29 AMLooks like you have an ancient version of singularity. Possibly if you have a module system there are newer versions available there?
Otherwise, both singularity and apptainer can be installed as a user with sensible functionality (mounting of images rather than unpacking). Apptainer has a nice package to help doing an unprivileged installation.
And for most setups, rather than have a job run all processes for nextflow, you want to have each nextflow process run as a separate job (this means teaching nextflow about your scheduler if there isn't a profile available for your system). The main nextflow process (I usuallly refer to it as monitoring) have very modest requirements and unless your system has really strict rules about login/submission node usage, I usually recommend leaving it there (typically in a `screen`/`tmux`) rather than wasting resources with a job for it.
2 Views