Hello, not 100% sure where is best to post this, I'm currently building a pipeline for our lab for genome assembly. I've been running into an issue with an nf-core module pairtools/sort. When i use this tool outside of nf-core pipeline it would use max 64GB, however inside the nf-core module i'm having lots of out of memory errors, the latest one tried using up to 600GB just for this single process. I'm running this on pawsey using singularity. not 100% on who to ask about this, its not a pairtools issue as i can run it fine outside of the nf-core workflow, but i'm not sure how it could be a nextflow issue either... if anyone has any ideas please let me know

Job ID: 15064447
Cluster: setonix
User/Group:
State: OUT_OF_MEMORY (exit code 0)
Nodes: 1
Cores per node: 128
CPU Utilized: 01:28:14
CPU Efficiency: 1.78% of 3-10:35:44 core-walltime
Job Wall-clock time: 00:38:43
Memory Utilized: 601.45 GB
Memory Efficiency: 150.36% of 400.00 GB

if [[ -n /scratch/pawsey0812/lhuet ]]; then
    temp=/scratch/pawsey0812/lhuet
fi
pairtools \
    sort \
    --nproc 32 \
    --nproc 64 \
    --tmpdir=/scratch/pawsey0812/lhuet \
    --memory 400G \
    -o OG647_v240501.hic1.pairs \
    OG647_v240501.hic1.pairsam > log.txt

cat <<-END_VERSIONS > versions.yml
"OCEANGENOMES_REFGENOMES:REFGENOMES:OMNIC_HAP1:PAIRTOOLS_SORT":
    pairtools: $(pairtools --version 2>&1 | tr '\n' ',' | sed 's/.*pairtools.*version //' | sed 's/,$/\n/')
END_VERSIONS

slurmstepd: error: Detected 1 oom-kill event(s) in StepId=15064447.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

When you mention what the tools does outside of nextflow, are you using the same parameters? Are you using the same binary (singularity image)? What resources were allocated to that job (15064447)?

outside of the nextflow i was running a bash script with the same image used in the container. The bash script had an entire library complexity analysis pipeline in it using the following script:

#!/bin/bash --login
#---------------


#Requested resources:
#SBATCH --account=pawsey0812
#SBATCH --job-name=HICLCA
#SBATCH --partition=work
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --time=15:00:00
#SBATCH --mem=64G
#SBATCH --export=ALL
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err


date=$(date +%y%m%d)


echo "========================================="
echo "SLURM_JOB_ID = $SLURM_JOB_ID"
echo "SLURM_NODELIST = $SLURM_NODELIST"
echo "DATE: $date"
echo "========================================="


full_path=$(pwd)
echo "Full path: $full_path"


run_dir="$1"
echo "Run directory: $run_dir"
sample_dir="$2"
echo "Sample directory: $sample_dir"
OG="$3"
echo "Specimen ID: $OG"
 
# Define path to temp directory
temp="$4"


 #Define Hi-C files
H1="${run_dir}/${sample_dir}/"*R1*.fastq.gz


for file in $H1; do
    echo "Hi-C forward: $file"
done


H2="${run_dir}/${sample_dir}/"*R2*.fastq.gz


for file in $H2; do
    echo "Hi-C reverse: $file"
done


asm="${run_dir}/${sample_dir}/"*.fasta


echo assembly: $asm


singularity run $SING/samtools_1.16.1.sif samtools faidx $asm && \
cut -f1,2 "${run_dir}/${sample_dir}/"*.fai > "${run_dir}/${sample_dir}/${OG}.genome" && \
singularity run $SING/bwa:0.7.17.sif bwa index $asm && \
singularity run $SING/bwa:0.7.17.sif bwa mem -5SP -T0 -t16 $asm $H1 $H2 -o "${run_dir}/${sample_dir}/${OG}.aligned.sam" && \

singularity run $SING/pairtools:1.1.0.sif pairtools parse --min-mapq 40 --walks-policy 5unique --max-inter-align-gap 30 --nproc-in 16 --nproc-out 16 --chroms-path "${run_dir}/${sample_dir}/${OG}.genome" "${run_dir}/${sample_dir}/${OG}.aligned.sam" > "${run_dir}/${sample_dir}/${OG}.parsed.pairsam" && \
singularity run $SING/pairtools:1.1.0.sif pairtools sort --tmpdir=$temp --nproc 16 "${run_dir}/${sample_dir}/${OG}.parsed.pairsam" > "${run_dir}/${sample_dir}/${OG}.sorted.parisam" && \
singularity run $SING/pairtools:1.1.0.sif pairtools dedup --nproc-in 16 --nproc-out 16 --mark-dups --output-stats "${run_dir}/${sample_dir}/${OG}.stats.txt" --output "${run_dir}/${sample_dir}/${OG}.dedup.pairsam" "${run_dir}/${sample_dir}/${OG}.sorted.pairsam" && \
singularity run $SING/pairtools:1.1.0.sif pairtools split --nproc-in 16 --nproc-out 16 --output-pairs "${run_dir}/${sample_dir}/${OG}.mapped.pairs" --output-sam "${run_dir}/${sample_dir}/${OG}.unsorted.bam" "${run_dir}/${sample_dir}/${OG}.dedup.pairsam" && \
singularity run $SING/samtools_1.16.1.sif samtools sort -@16 -T $temp.bam -o "${run_dir}/${sample_dir}/${OG}.mapped.PT.bam" "${run_dir}/${sample_dir}/${OG}.unsorted.bam" && \
singularity run $SING/samtools_1.16.1.sif samtools index "${run_dir}/${sample_dir}/${OG}.mapped.PT.bam"


wait 

singularity run $SING/pairtools:1.1.0.sif python3 get_qc.py -p "${run_dir}/${sample_dir}/${OG}.stats.txt" && \
singularity run $SING/preseq:2.0.2.sif preseq lc_extrap -bam -pe -extrap 2.1e9 -step 1e8 -seg_len 1000000000 -output "${run_dir}/${sample_dir}/${OG}.out.preseq" "${run_dir}/${sample_dir}/${OG}.mapped.PT.bam"

in nf-core i have created a sub workflow for this so each is its own process. for the pairtools sort on that last run i had allocated the following resources after some failures at 64GB of memory

withName:PAIRTOOLS_SORT {
        queue = 'highmem'
        cpus   = { 64 }
        memory = { 400.GB }
        time   = { 14.h }

withName: 'PAIRTOOLS_SORT' {
        ext.args = { '--nproc 32' }
    publishDir = [
        path: { "${params.outdir}/${meta.sample}/04-scaffolding/omnic" }
        ]
    }

I don't know this tool, but it's not impossible you many need to scale them separately (e.g. try with like 4 cpus rather than 64). It's of course also possible that the way the module calls it doesn't really make sense (and e.g.

--memory

should be removed), but as a first suggestion, I'd try not scaling both the number of parallel processes and memory.

ill test scaling down the cpus and see how it goes

For reference, the nf-core modules code: https://github.com/nf-core/modules/blob/f6cc00f107826cfaf1c933297b10ed1757b41479/modules/nf-core/pairtools/sort/main.nf this has:

def buffer = task.memory.toGiga().intdiv(2)
    """
        --nproc $task.cpus \\
        --memory ${buffer}G \\

so you don't need to specify

nproc

through in the ext.args.

this divides the memory by 2 .intdiv(2), do we know why that is?

https://github.com/nf-core/modules/pull/3909 (whatever that's good or not, I don't have an opinion on, typically I wouldn't expect

SIGTERM

for out of memory, but it's not impossible that's sent first and handled or that there's some manual watchdog in their setup, killing processes nicely.

It is common to leave some memory space, because running docker/singularity requires some overhead