Hello all, I've quite the same kind of issue than @Michael Martinez. I'm trying to run the smrnaseq pipeline ver 2.4.1. I've used in the past old version of the pipeline. Currently I'm stuck, because I keep getting the

Process requirement exceeds available memory

message.

ERROR ~ Error executing process > 'NFCORE_SMRNASEQ:UMICOLLAPSE_FASTQ (MO2025_03_2_07)'

Caused by:
  Process requirement exceeds available memory -- req: 200 GB; avail: 125.5 GB


Command executed:

  # Getting the umicollapse jar file like this because `umicollapse` is a Python wrapper script generated
  # by conda that allows to set the heap size (Xmx), but not the stack size (Xss).
  # `which` allows us to get the directory that contains `umicollapse`, independent of whether we
  # are in a container or conda environment.
  UMICOLLAPSE_JAR=$(dirname $(which umicollapse))/../share/umicollapse-1.1.0-0/umicollapse.jar
  java \
      -Xmx184320M \
      -Xss999M \
      -jar $UMICOLLAPSE_JAR \
      fastq \
      -i MO2025_03_1_08_S6_R1_001.fastq.gz \
      -o MO2025_03_1_08.umi_dedup.sorted.fastq.gz \
      --algo dir --two-pass | tee MO2025_03_1_08.umi_dedup.sorted_UMICollapse.log
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_SMRNASEQ:UMICOLLAPSE_FASTQ":
      umicollapse: 1.1.0-0
  END_VERSIONS

Command exit status:
  -

Command output:
  (empty)

I've used, and still using, quite a large number of nf-core pipeline. On different workstations (still waiting for an installation on our HPC). I've had quite a lof of issues and bugs in the past. But these error messages are new for me. I've tried to modify by hand the UMICOLLAPSE:main.nf in order to hard coded the allocated memory. No luck. I've put

def max_heap_size_mega = 20000

and the process keeps asking for 200 Gb.

ERROR ~ Error executing process > 'NFCORE_SMRNASEQ:UMICOLLAPSE_FASTQ (MO2025_03_2_07)'

Caused by:
  Process requirement exceeds available memory -- req: 200 GB; avail: 125.5 GB


Command executed:

  # Getting the umicollapse jar file like this because `umicollapse` is a Python wrapper script generated
  # by conda that allows to set the heap size (Xmx), but not the stack size (Xss).
  # `which` allows us to get the directory that contains `umicollapse`, independent of whether we
  # are in a container or conda environment.
  UMICOLLAPSE_JAR=$(dirname $(which umicollapse))/../share/umicollapse-1.1.0-0/umicollapse.jar
  java \
      -Xmx20000M \
      -Xss999M \
      -jar $UMICOLLAPSE_JAR \
      fastq \
      -i MO2025_03_2_07_S11_R1_001.fastq.gz \
      -o MO2025_03_2_07.umi_dedup.sorted.fastq.gz \
      --algo dir --two-pass | tee MO2025_03_2_07.umi_dedup.sorted_UMICollapse.log

My nextflow is up-to-date and I've dowloaded the pipeline on the same computer.

nextflow -v
nextflow version 25.10.2.10555

I'm using the following command:

nextflow run \
/mnt/ssd/Nextflow/nf-core-smrnaseq/2_4_1/ \
-profile singularity \
-params-file /XXXXXXXXXXX/data/params.yml

with the parameters:

input: "/XXXXXXXXXXX/data/samplesheet.csv"
outdir: "/XXXXXXXXXXX/results/"
protocol: "qiaseq"
mirgenedb: true
mirgenedb_species: "rno"
mirgenedb_gff: "/XXXXXXXXXXX/mirgene/rno.gff"
mirgenedb_mature: "/XXXXXXXXXXX/mirgene/rno-mature.fas"
mirgenedb_hairpin: "/XXXXXXXXXXX/mirgene/rno-pre.fas"
genome: Rnor_6.0
fasta: /XXXXXXXXXXX/references/Rattus_norvegicus/Ensembl/Rnor_6.0/Sequence/WholeGenomeFasta/genome.fa"
bowtie_index: "/XXXXXXXXXXX/references/Rattus_norvegicus/Ensembl/Rnor_6.0/Sequence/BowtieIndex"
with_umi: true
umitools_extract_method: "regex"
umitools_bc_pattern: '.+(?P<discard_1>AACTGTAGGCACCATCAAT){s<=2}(?P<umi_1>.{12})(?P<discard_2>.*)'
skip_fastp: true
multiqc_title: "MultiQC Projet XXXXX - analyses miRNA"
filter_contamination: true
trna: "/XXXXXXXXXXX/tRNA/rn6-mature-tRNAs.fa"
cdna: "/XXXXXXXXXXX/fasta/Rattus_norvegicus.GRCr8.cdna.all.fa"
ncrna: "/XXXXXXXXXXX/fasta/Rattus_norvegicus.GRCr8.ncrna.fa"

Does that mean my max avalaible memory is hard coded somewhere?

With nf-core pipelines, you should essentially never modify the pipeline code.

First, thank you for your message. Ok. Yesterday, before your answer, I modified the file

base.config

where the memory for some process was hardcoded with a value incompatible with my workstation:

withLabel:process_high_memory {
        memory = { 200.GB * task.attempt }

it was working afterwards. Strangely I never had this issue before, even though over pipelines I used have the same configuration file. Perhaps, this happened after my update of Nextflow binairies. I tried this morning the solution you linked me. It's working and it is more "nextflow"-elegant. So adopted.

Ok, see "Tuning workflow resources" (the next section from the link above) for the nf-core way of managing that.