Mehrdad Jaberi
05/09/2025, 9:14 AMmeta, and let downstream modules decide resources on the fly—e.g. `cpus { meta.reads < 1e7 ? 2 : 8 } memory { meta.reads < 1e7 ? 6.GB : 32.GB } time { meta.reads < 1e7 ? 4.h : 24.h } errorStrategy { task.attempt < 2 ? 'retry' : 'finish' }`—so small libraries finish fast, big ones get what they need, retries stay minimal, and we leverage data we already collect without extra commands.
or resource request could use a look up table.Phil Ewels
05/09/2025, 9:16 AMPhil Ewels
05/09/2025, 9:16 AMPhil Ewels
05/09/2025, 9:17 AMPhil Ewels
05/09/2025, 9:19 AMPhil Ewels
05/09/2025, 9:20 AMSimon Pearce
05/09/2025, 9:22 AMThis has come up a few times before. Agree that it sounds very attractive, the problem is that calculation of things like read length etc. requires reading the data.Yes, hence the suggestion to extract them via process that may well be already doing. Alternatively, can we get the size of the input files (in Gb)?
Phil Ewels
05/09/2025, 9:28 AMPhil Ewels
05/09/2025, 9:28 AMMehrdad Jaberi
05/09/2025, 9:28 AMPhil Ewels
05/09/2025, 9:29 AMPhil Ewels
05/09/2025, 9:30 AMPhil Ewels
05/09/2025, 9:30 AMMehrdad Jaberi
05/09/2025, 9:33 AMSimon Pearce
05/09/2025, 9:35 AMSimon Pearce
05/09/2025, 9:36 AMJames Fellows Yates
05/09/2025, 9:36 AMJames Fellows Yates
05/09/2025, 9:37 AMJames Fellows Yates
05/09/2025, 9:40 AMmodules.conf no?James Fellows Yates
05/09/2025, 9:41 AMSimon Pearce
05/09/2025, 9:46 AMJames Fellows Yates
05/09/2025, 9:47 AMModules MUST NOT use ‘custom’ hardcoded meta fieldsJames Fellows Yates
05/09/2025, 9:47 AMSimon Pearce
05/09/2025, 9:47 AMJames Fellows Yates
05/09/2025, 9:47 AMJames Fellows Yates
05/09/2025, 9:48 AMJames Fellows Yates
05/09/2025, 9:48 AMJames Fellows Yates
05/09/2025, 9:49 AMJames Fellows Yates
05/09/2025, 9:49 AMSimon Pearce
05/09/2025, 9:49 AMJames Fellows Yates
05/09/2025, 9:50 AMSimon Pearce
05/09/2025, 9:51 AMJames Fellows Yates
05/09/2025, 9:52 AMMatthieu Muffato
05/09/2025, 10:08 AMFinding the right formula isn’t always trivial. Even for read processors, resource usage may depend on the read length (especially relevant if you want to support all flavours of Illumina and ONT and PacBio etc), (in)directly on the species’ genome size, number of chromosomes, etc if the reads are mapped. Some tools do clever method for like sorting/grouping/aligning reads, which makes their resource usage in fact dependent on the polymorphism rate (which is relevant to support multiple organisms). Thecpus { meta.reads < 1e7 ? 2 : 8 } memory { meta.reads < 1e7 ? 6.GB : 32.GB } time { meta.reads < 1e7 ? 4.h : 24.h } errorStrategy { task.attempt < 2 ? 'retry' : 'finish' }
time constraint depends on the CPUs you’ve got access to, which can be multiple times faster across generations, on the disk I/O rate which can also vary greatly if you think local disk vs network disk, etc.
It’s a great idea, and we try to do it in our pipelines, but one needs to be very careful about extrapolating the rules and having something that’s good for everyone/enough people and suitable for inclusion in nf-core/modules.Mehrdad Jaberi
05/09/2025, 10:27 AMprocess FASTQC {
input: tuple val(id), path(fq)
output: tuple val(id), path(fq), val(reads)
}
process TRIM {
input: tuple val(id), path(fq), val(reads)
cpus { reads < 1e7 ? 2 : 6 }
output: tuple val(id), path("trimmed.fq.gz"), val(reads)
}
process MAP {
input: tuple val(id), path(trimmed), val(reads)
memory { reads < 1e7 ? 8.GB : 32.GB }
...
}James Fellows Yates
05/09/2025, 11:00 AMmodules.config file.
Furthermore, by having it in a config file that will be a way of setting such things that is more familiar for users who would have to more likely make a custom config themselves toadapt to their data.
So for me:
1. No to embedding in the modules
2. Yes for finding ways to facilitate such a concept in as a user friendly system as possible that both users/pipeline devs can configure themselves :)Phil Ewels
05/09/2025, 12:02 PMJames Fellows Yates
05/09/2025, 12:03 PMJames Fellows Yates
05/09/2025, 12:03 PMenv outputs and then pulled into a meta in the pipeline?Phil Ewels
05/09/2025, 12:04 PMenv bits but yeah..James Fellows Yates
05/09/2025, 12:04 PMArthur
05/10/2025, 12:34 AM