Hi team, I wonder if someone could help me out with the rnasplice pipeline (https://nf-co.re/rnasplice/1.0.3). I'm trying to run this on AWS batch, but keep constantly getting out of memory errors (especially for samtools sort and sometimes index). I tried to change the config file (aws.config) to allow more memory for these processes, to specify number of cores, and to make it serialized instead of parallel, as follows:

aws {

batch {

volumes = ['/scratch:/tmp']

}

}

process {

maxRetries = 3

withName: 'NFCORE_RNASPLICE:RNASPLICE:ALIGN_STAR:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT' {

queueSize = 1

memory = '64 GB'

cpus = 4

}

withName: 'NFCORE_RNASPLICE:RNASPLICE:ALIGN_STAR:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_INDEX' {

queueSize = 1

memory = '64 GB'

cpus = 4

}

}

However, the error message still shows:

Caused by:

Essential container in task exited - OutOfMemoryError: Container killed due to memory usage

Command executed:

samtools sort \

\

-@ 4 \

-m 16384M \

-o P633L_3_sorted.bam \

-T P633L_3_sorted \

P633L_3.Aligned.out.bam

cat <<-END_VERSIONS > versions.yml

"NFCORE_RNASPLICE:RNASPLICE:ALIGN_STAR:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT":

samtools: $(echo $(samtools --version 2>&1) | sed 's/^.*samtools //; s/Using.*$//')

END_VERSIONS

Command exit status:

137

Command output:

(empty)

Command error:

.command.sh: line 8:   121 Killed                  samtools sort -@ 4 -m 16384M -o P633L_3_sorted.bam -T P633L_3_sorted P633L_3.Aligned.out.bam

Notice the 16GB on the samtools sort command. Here's how I run this:

nextflow -bg run nf-core/rnasplice -profile awsbatch -r 1.0.3 -work-dir <s3://ICANTSHARETHAT/work> -with-report <s3://ICANTSHARETHAT/report.html> -params-file paramseb.yaml -with-trace <s3://ICANTSHARETHAT/trace.txt> -c /home/ec2-user/scratch/aws.config

Here's the paramseb.yaml (omitting s3 locations for input, output, genome files, etc):

sashimi_plot: true

miso_genes: ENSG00000145349.20, ENSG00000143079.15

miso_read_len: 193

awsqueue: something-large

awsregion: us-east-1

awscli: /home/ec2-user/miniconda/bin/aws

rmats: true

rmats_read_len: 193

suppa: true

suppa_per_local_event: true

local_events: SE SS MX RI FL

dexseq_dtu: false

Any idea on what I could do differently to allow more memory to samtools? I really appreciate any help you can provide. Thanks for an excellent tool!

I looked at the source code, the 16MB for the parameter is exactly what you would expect:

def sort_memory = (task.memory.mega/task.cpus).intValue()

task.memory

= 64 GB,

task.cpus

= 4, 64 GB/4 = 16 GB per core. So if this a pipeline issue, you could give it even more RAM or decrease the CPUs and see if that helps. If this something specific to AWS batch, however, I recommend asking in #infrastructure-aws over in the Nextflow Slack.

Thank you so much for taking the time to help. I'm increasing the memory as suggested.