Hello community. I'm not sure if what I'm going to ask makes sense, but after reading the Nextflow and nf-core documentation, I haven't been able to find a solution. I'm working on an HPC environment. I'm runnin rnaseq. If I run this command directly in the bash, it executes without any problem.

nextflow run . --input samplesheet.csv --oudir test_results --genome GRCh38 --profile singularity -c config.config

In the

config.config

file, I specify that I want it to run with SLURM and provide the necessary configuration. (My institution does not have a .config file)

process {
  executor = 'slurm'
  queue = 'short'
  cpus = 2
  memory = '10.GB'
  time = '4h'

  resourceLimits = [
    cpus: 32,
    memory: '100.GB',
    time: '24h'
  ]
  withLabel:process_high {
    cpus = 8
    memory = '32.GB'
    time = '12h'
  }
}
executor.perCpuMemAllocation = true

However, when I run it this way, the process works within the session but prevents me from continuing to work or disconnecting. What I want to do is create an sbatch script and submit the job to SLURM. Unfortunately, when I run this sbatch, it doesn't work, and the job ends after about 30 seconds.

#!/bin/bash

#SBATCH -J RNASEQ
#SBATCH -o output.rnaseq.out
#SBATCH -e error.rnaseq.err
#SBATCH -c 2
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem=10000M

# Load Nextflow
module load Nextflow/24.04.2

# Run analysis
nextflow run . --input samplesheet.csv --oudir test_results --genome GRCh38 --profile singularity -c config.config

Is there any configuration I need to consider before submitting the sbatch to SLURM? Thank you very much!

Do you need to

module load

singularity` on the compute node? But otherwise it's hard to say what could be causing it without seening the error message

Agreed, we may have psychic debugging powers but there are limits to everything. Also, depending on site policy, it may make more sense to run for nextflow monitoring process on the login/submitting node in a

tmux

(or

screen

). Nextflow can also be run in the background if that makes more sense.

Thank you both. I don't need to

module load

the Singularity module on that HPC (and I didn't do it when running in Bash either — I only do it with Nextflow). The error says:

N E X T F L O W  ~ version 24.04.2

ERROR ~ Unable to parse config file: '/gpfs3/well/church/users/usj491/nf-core-rnaseq/3_18_0/nextflow.config'

 Cannot read config file include: <https://raw.githubusercontent.com/nf-core/configs/master/nfcore_custom.config>

But since I'm using exactly the same command that works in Bash:

nextflow run . --input samplesheet.csv --oudir test_results --genome GRCh38 --profile singularity -c config.config

…the problem must be in how SLURM is handling my configuration, right? TA!

I guess that could depend on what you mean with "how SLURM is handling my configuration" (it doesn't 🙂 But that looks like you have a connectivity issue. Do compute nodes have internet access?

Thank you @pontus. I think you're right. The nodes have not internet access, but the login node does. So that's why it's ending. So do you recommend me to use

txum

and forget SLURM?

Typically this means yes you should run the main `nextflow run`commabd in tmux on the headnode, but you must ensure you've downloaded any required databases/reference genomes in advance (some pipelines will try to download this for you)

Brilliant. Thank you very much @pontus @James Fellows Yates. Thank you for your help!