https://nf-co.re logo
Join Slack
Powered by
Hi James, I am starting a new thread to ask the "b...
# mag
r
Hi James, I am starting a new thread to ask the "best practice" regarding requesting CPU and memory via slurm on HPC: I am bothered by increased CPU and memory a lot recently and I was able to avoid that as assembly steps using ``--spades_fix_cpus`` and ``--megahit_fix_cpu_1``. However, the subsequent steps are also giving me this trouble as shown in the examples below. Looking into this, I saw that required memory and cpu is associated with task.attempt:
Copy code
withName: GTDBTK_CLASSIFYWF {
    cpus   = { 10 * task.attempt }
    memory = { 128.GB * task.attempt }
    time   = { 12.h * task.attempt }
}

withName: METASPADES {
    cpus          = { params.spades_fix_cpus != -1 ? params.spades_fix_cpus : (10 * task.attempt) }
    memory        = { 64.GB * (2 ** (task.attempt - 1)) }
    time          = { 24.h * (2 ** (task.attempt - 1)) }
    errorStrategy = { task.exitStatus in (...) ? 'retry' : 'finish' }
    maxRetries    = 5
}
since I will need to request specific number of cpus and amount of memories using slurm, what would you recommend to do in this case? Keep increasing mem/cpu when it fails? is there a better way?
Copy code
Workflow execution completed unsuccessfully!
The exit status of the task that caused the workflow execution to fail was: null.

The full error message was:

Error executing process > 'NFCORE_MAG:MAG:ASSEMBLY:SHORTREAD_ASSEMBLY:METASPADES (CR180008)'

Caused by:
  Process requirement exceeds available memory -- req: 128 GB; avail: 120 GB


Command executed:

  spades.py \
      --meta \
      --threads 48 \
      --memory 128 \
       \
      -1 CR180008_run0_phix_removed.unmapped_1.fastq.gz -2 CR180008_run0_phix_removed.unmapped_2.fastq.gz   \
      -o ./
  mv spades.log SPAdes-CR180008.spades.log
Copy code
Workflow execution completed unsuccessfully!
The exit status of the task that caused the workflow execution to fail was: null.

The full error message was:

Error executing process > 'NFCORE_MAG:MAG:GTDBTK:GTDBTK_CLASSIFYWF (MEGAHIT-MaxBin2-unclassified-unrefined-CR180008)'

Caused by:
  Process requirement exceeds available memory -- req: 128 GB; avail: 120 GB


Command executed:

  export GTDBTK_DATA_PATH="$(find -L gtdb -name 'metadata' -type d -exec dirname {} \;)"
  
  if [ "--scratch_dir pplacer_tmp" != "" ] ; then
      mkdir pplacer_tmp
  fi
  
  gtdbtk classify_wf \
      --extension fa --min_perc_aa 10 --min_af 0.65 --pplacer_cpus 1   \
      --genome_dir bins \
      --prefix "MEGAHIT-MaxBin2-unclassified-unrefined-CR180008" \
      --out_dir MEGAHIT-MaxBin2-unclassified-unrefined-CR180008 \
      --cpus 10 \
      --scratch_dir pplacer_tmp
and my slurm script:
Copy code
[liruiqi@discovery2 docker_nohost]$ cat run_nfcore_mag_apptainer.slurm
#!/bin/bash
#SBATCH --job-name=nfcore_app
#SBATCH --partition=main
#SBATCH --time=18:00:00
#SBATCH --cpus-per-task=16
#SBATCH --mem=120G
#SBATCH --output=/project2/ckenkel_26/liruiqi/mag_out_project2/logs/nfcore_mag.%j.out
#SBATCH --error=/project2/ckenkel_26/liruiqi/mag_out_project2/logs/nfcore_mag.%j.err
#SBATCH --mail-user=liruiqi@usc.edu
#SBATCH --mail-type=END,FAIL

# Move to project directory
cd /project2/ckenkel_26/liruiqi/mag_out_project2/docker_nohost
module purge
module load apptainer/1.4.5
# Activate conda environment
source ~/.bashrc
conda activate meta

# Run nf-core/mag
nextflow run nf-core/mag \
  -profile apptainer \
  --input /project2/ckenkel_26/liruiqi/mag_out_project2/samplesheet.csv \
  --outdir /project2/ckenkel_26/liruiqi/mag_out_project2/docker_nohost \
  --spades_fix_cpus 16 \
  --gtdb_db /project2/biodb/gtdb \
  --host_fasta /project2/ckenkel_26/liruiqi/mag_out_project2/Osedax_frankpressi.fna \
  --host_removal_verysensitive
j
resourcehelp
s
You likely need to configure the pipeline to your available computational resources to you machine or infrastructure. Please see the following two(!) sections: - https://nf-co.re/docs/usage/getting_started/configuration#tuning-workflow-resources - https://nf-co.re/docs/usage/getting_started/configuration#max-resources of the documentation for more information.
j
I would suggest reading the two pages above, this should hopefully explain how to configure correctly a pipeline to your available resource
And if you're on a HPC, to properly get the maximium benefit you want to write a configuration file
Note you don't have to submit to the nf-core/configs central repository if you don't want to
but the concept still standas
r
Thank you!!!
5 Views
Open in Slack
PreviousNext
Powered by