On a scale from 1-10 how awful do we think this is? https://github.com/artic-network/amplicon-nf/blob/6da2914720de5076da0f003b4d5304fd6360193a/nextflow.config#L92-L110 epi2melabs still runs nxf version 23.04.2 and this approach to deal with the lack of resourceLimits in a way that doesn't break compatibility with current versions genuinely came to me in a dream

You can clean it up slightly by doing

withLabel: "*"

if you like

Oooooh that's a good idea

What error does that cause?

image-2.png

ah sorry, @Arthur might be right on that one!

Potential issues I see: • Your resource limits are set to 7 GB and 2 CPUs. However, if a process fails and gets rerun on the old Nextflow, it will do so with 14 GB and 4 CPUs (or 21 GB and 6 CPUs the 2nd time), so depending what your actual limits are, this could be a problem. To make the

resourceLimits

and

withLabel

statements equivalent, you would have to to remove the

* task.attempt

. ◦ The older versions of nf-core pipelines used

--max_cpus

and

--max_memory

parameters in conjunction with a Groovy function to cap the values passed to

cpus

and

memory

in the config, so that

* task.attempt

could be used without blowing the limit, • Also, I'm not sure I like that resource limits are set in a specific profile inside the

nextflow.config

. Any kind of limits are typically driven by your computing setup, not really by the pipeline itself, so they should be in a separate config file (like these). The way you have it now, you would will always need to run with

-profile standard

and potentially add other profiles, like

debug

and/or

conda

on top of that, but if you have contradictory statements in combined profiles (like

docker = ...

), that's problematic. If you really want to set

resourceLimits

(and the equivalent

withLabel

statements) at all in

nextflow.config

, I would do that outside all profiles, so that it always applies, no matter which profile you use.

All reasonable points but this is specifically for running in epi2melabs so standard is the only way to get any sort of irregular config since it doesn't support profiles, the usecase is for people who have zero command line experience to be able to use the pipeline especially people in LMICs with really bad old laptops hence 2 / 7 requests, the rest of the pipeline is much more sanely laid out

Well, you know your use cases and what is possible better than anyone else, but my instinct would be to put anything that is specific to a particular use case (e.g. inexperienced users with old laptops) into some type of wrapper script and tell them to use that.