Congrats to @Felix Krueger for getting Trim Galore v2.0 rewrite released 🎉 https://www.trimgalore.com/

+ the nf-core module already, nice!!!

Hah, I didn't see that yet.. @Jonathan Manning is a machine! 🚀 😆 https://github.com/nf-core/modules/pull/11524

It’s also already in nf-core/rnaseq! Amazing work altogether!! Thanks, @Jonathan Manning!!

🤯 https://github.com/nf-core/rnaseq/pull/1789

Anybody who'd like to see it in a release should review my PRs in #CQY2U5QU9 🙂

🔥 🔥 🔥

We should probably take requested resources down to 100MB of RAM and 10 min runtime 🙂

We probably need to move it to a lower label- PRs coming...

First real data run with

dev

(~30M PE per sample):

Cool! Modules PR: https://github.com/nf-core/modules/pull/11531 I think that's the smallest sensible bucket, but runtime optimised resource configs will be useful to not over-provision 🙂

I just added this comment: As a comment on this, the invocation was

--cores 8

,(rna-seq default), which uses: • ~12 CPUs • ~100MB of RAM • with roughly 20M read pairs per minute Please note that the threading model for parallel runs is

N+4

, so using

--cores 2

uses 6 cores, while

--cored 8

uses 12 cores. Using

--cores 2

is still good time wise, but the wall clock time can be reduced by a further 75% by using 12 CPUs (

--coreds 8

).

Yep, but the nf-core standards currently resolve around these labels, so need to pick the 'good enough' one there.

so does it then run in single CPU mode, given that even

--cores 2

would require 6CPU?

I'm not sure. I don't really follow the logic of having the user request 2 cores but requiring 6, seems like asking for trouble.

Yes, this has always been the case with @Felix Krueger’s tools 😅

well, it follows a formula of N+4….:

Oxidized Edition --cores N thread breakdown:
  N worker threads (each: trim + gzip compress -> independent gzip block)
  2 decompression threads (one per input file)
  1 batcher thread (creates numbered batches of 4096 reads)
  1 main thread (collects blocks in order -> writes to output files)
                                                       = N+4 total

to me,

--cores 2

is 2x as fast,

--cores 4

is 4x as fast,

--cores 8

is 8x as fast. That kind of works. Adding the 4 overhead threads kind of breaks this logic (but comes at the expense that

label low

doesn’t quite work… to be honest to me this is a pipeline setting, and not a TG setting: set it once, done.

So 2 cores is reasonable for nf-core/modules? We can add custom config in rnaseq, this is just a non-mad default, and we only have the existing bins to work with right now: https://github.com/nf-core/rnaseq/blob/master/conf/base.config FWIW nobody is going to understand the +4 rule, sorry!

I think

--cores

should specify how many cores will be used 👀

I can’t really help that the current low/medium/high labels don’t quite fit TG oxy anymore….

process_high

has the correct amount of CPUs for maximum performance, but it asks for 720 times (!) more memory than is needed. I see 4 options, in terms of usefulness: 1. introduce a new label for memory and time efficient tools (such as

fastqc-rust

,

trim-galore

) but have decent CPU parallelisation required for maxing efficiency and reduction in wall clock time (see above). This is what I would expect as the default as a user 2. do not assign a ‘label’ to the Trim Galore module at all, but move cpus = { 12 * task.attempt }, memory = { 1.GB * task.attempt }etc. into the module instead. This bypasses adding a new general label to base config but still be optimal for the tool in question. This is equivalent to 1 in that it is optimised, but kind of goes against the old base config rules) 3. keep

process_high

but ask folks to each add a new config (or use groundswell) to ask for 700 times less memory). Burdensome and goes against the assumption that default == good default 4. keep

process_low

but accept that

--cores 2

uses 6 CPUs, and thus overcommit the node dramatically. Not desirable

Can just do

config.cores = cli.cores - 4

if needed? Easier to handle that logic internal to the tool rather than putting it onto the end user to figure out

completely agree that this should not be up to the end user to figure out. It’s a one time setting to get defaults right (and maximally efficient)

1. is not as easy as you might think, it has wide implications in nf-core, will just take some discussion and I'm going to put out a release ASAP 🙂 2. would be very non-standard, not sure I'd get it past- we work with labels on the modules repo 3. I don't think a default of that much over-provisioning is sensible. I think 4 is the least-worst default in the modules repo, and we can override that in the rnaseq config.

Hmm, not sure if asking for 2 CPUs but using 6 is least worst 🙂. If you want to get it out without hassle it is probably easiest to go with

--process_high

as this is most efficient setting. RAM can’t cost all that much? (overpaying >> over-commiting?)

I think as a user I'd be more annoyed about a tool asking for 72Gb RAM when it needs <1Gb than I would be about a little sub-optimal performance, and it's only a default, people can override the resources when they deploy to their workflow (as we will here).

If we do it here, it will be fine, but it introduces a burden for every over workflow that they all need the same fix for over-commiting default settings. Not a blocker for the current release, but I think it warrants a discussion goingforward medium would certainly better

I wonder if we just need a new

process_low_memory

label 🤔

process_high

followed by

process_low_memory

would do the trick just fine

I think we should revisit the idea of having low, medium, high labels for CPU, time and memory separately. I know that the initial argument against was that both on HPC and Cloud machine instances, memory and CPUs are typically coupled. However, most cloud providers now have a large selection of different instances. Even when coupled, there is no one size fits all solution.

https://github.com/nf-core/tools/pull/4264 for the minimum.

Throwing some oil on the fire - it is often OK to oversubscribe a CPU but oversubscribed memory usually leads to paging which is extremely detrimental to performance.