Hi all. who is known with this error? is this because of HPC cluster, or pipeline?

ERROR ~ Error executing process > 'NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM (EGAN00001378824-lane2)'

Caused by:
  Process `NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM (EGAN00001378824-lane2)` terminated with an error exit status (1)

Command executed:

  INDEX=`find -L ./ -name "*.amb" | sed 's/\.amb$//'`
  
  bwa mem \
      -K 100000000 -Y -R "@RG\tID:HNKLJDMXY.EGAN00001378824.lane2\tPU:lane2\tSM:EGAN00001378824_EGAN00001378824\tLB:EGAN00001378824\tDS:<s3://ngi-igenomes/igenomes/Homo_sapiens/GATK/GRCh38/Sequence/WholeGenomeFasta/Homo_sapiens_assembly38.fasta>\tPL:ILLUMINA" \
      -t 24 \
      $INDEX \
      0001.EGAN00001378824-lane2_1.fastp.fastq.gz 0001.EGAN00001378824-lane2_2.fastp.fastq.gz \
      | samtools sort  --threads 24 -o EGAN00001378824-lane2.0001.bam -
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM":
      bwa: $(echo $(bwa 2>&1) | sed 's/^.*Version: //; s/Contact:.*$//')
      samtools: $(echo $(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*$//')
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  +++ '[' 0 '!=' 0 ']'
  +++ for i in '"${!row[@]}"'
  +++ '[' 1 '!=' 0 ']'
  +++ sum[i]=0
  +++ for i in '"${!row[@]}"'
  +++ '[' 2 '!=' 0 ']'
  +++ sum[i]=28036
  +++ for i in '"${!row[@]}"'
  +++ '[' 3 '!=' 0 ']'
  +++ sum[i]=812
  +++ for i in '"${!row[@]}"'
  +++ '[' 4 '!=' 0 ']'
  +++ sum[i]=28040
  +++ for i in '"${!row[@]}"'
  +++ '[' 5 '!=' 0 ']'
  +++ sum[i]=6600
  +++ for i in '"${!row[@]}"'
  +++ '[' 6 '!=' 0 ']'
  +++ sum[i]=177
  +++ for i in '"${!row[@]}"'
  +++ '[' 7 '!=' 0 ']'
  +++ sum[i]=10
  +++ '[' 2 = 1 ']'
  +++ for i in '{1..7}'
  +++ '[' 0 -lt 7 ']'
  +++ sum[i]=7
  +++ for i in '{1..7}'
  +++ '[' 28036 -lt 5854292 ']'
  +++ sum[i]=5854292
  +++ for i in '{1..7}'
  +++ '[' 812 -lt 5799864 ']'
  +++ sum[i]=5799864
  +++ for i in '{1..7}'
  +++ '[' 28040 -lt 5854296 ']'
  +++ sum[i]=5854296
  +++ for i in '{1..7}'
  +++ '[' 6600 -lt 5799864 ']'
  +++ sum[i]=5799864
  +++ for i in '{1..7}'
  +++ '[' 177 -lt 330 ']'
  +++ sum[i]=330
  +++ for i in '{1..7}'
  +++ '[' 10 -lt 24 ']'
  +++ sum[i]=24
  +++ '[' 2 = 1 ']'
  +++ nxf_stat_ret=(${sum[*]})
  +++ '[' 4 -lt 10 ']'
  +++ timeout=1
  +++ read -t 1 -r DONE
  ++ kill 38

Work dir:
  /dss/work/auxu8708/work_rna/ae/6bdc98822b4c69a9def906b7979002

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details

Hi @Rafiga, Given the log you provided, my intuition is the pipeline has exceeded the provided resources and therefore, has been "killed". You can check the CPU (or thread) required, the memory usage, or the wall time, to see which one has caused the crash in your run.

I don't think it was killed by the scheduler, normally that would be an exit code >128, e.g. 140 for time (at least with SLURM). Exit code 1 suggests

bwa

encountered an error and quit. Where does that bizarre output on STDERR come from? It looks like from a shell script run with

-x

set? Is there any other error message in any of the

.command.*

files?

How could I control the consumption of resources?

If you think it's a resource issue, you can check with the job ID (the

.nextflow.log

should have a line looking like this:

... Task completed > TaskHandler[jobId: 123456; ...; status: COMPLETED; exit: 1; ...

. You can use that

jobId

with whatever tools your scheduler has (e..g

sacct

for SLURM) to query the resources requested and consumed. You should also find that information in the process table (use the search box to find your process) in the

execution_report.*.html

. If the consumed resources (specifically memory and time) go all the way to the request (or beyond), you'll need to request more (if available in your system; unless you can find a way to make the process use less, but it is rare that you can "control the consumption of resources", except for time you could try assigning more CPUs, but unless

bwa

has options that help with memory, you'll just need more, if that's the issue). The pipeline by default automatically reruns failed processed with double the initial resources (up to the max allowed, see the

--max_*

parameters) with certain exit codes, but

1

is not one of them, so you can modify

errorStrategy

to achieve that or assign this specific process more of whatever it has too little of (both in a config file).