Hi, i am a bit confused by the

bwamem2_index

module behaviour, it requests memory, based on

memory { 280.MB * _Math_.ceil(fasta.size() / 10000000) * task.attempt }

. However, on AWS it OOMs even on attempt 3. The log states:

ref seq len = 315317090

which I assume is the length of the sequence. By my math (

315317090*280/10000000

), this should request 8.828 MB * task.attempt, but according to the AWS metrics for this job it requested 3.3 GB for the third (1.1GB per attempt). This also was not an issue before, so I am a bit confused whats going wrong.

Check if

fasta

is

List

? or there is a variable

fasta

in that process.

fasta

is an input:

tuple val(meta), path(fasta)

Is that

path

a

List

object or a

Path

object?

Path

also implements the

List

interface too I think so iterators work on it, so it might be checking that

fasta.size()

is actually the file size and not the number of

/

+ 1

You mean the number of

/

in the path? That is either 0 (input filename) or 6 (full path). I think that this tries to estimate sequence length, based on filesize. Idk how reliable that is, I guess it is ok if sequence characters >>> header (which is the case). I also hope that

file.size()

checks the size of the of the source file, and not that of the symlink?

So you need a compression scaling factor then if zipped?

That is what I am guessing

Yeah, probably needs fixing in modules

I think it's a good idea. You can do a test on the file ending and then return triple the amount of memory if it ends in gz/bz2

The scaling * 3 is a wild guess tbh

I think it is just the bwa index family, as they have a well characterised dependency

@Niklas Schandry. gzip consistently compresses files ~3x, it’s fine to hardcode that

If I can figure out why i cannot test locally, and manage to fix it i will open a pr ;)

Yeah