Hi there, I was wondering if this could be someth...
# modules
m
Hi there, I was wondering if this could be something to maneuver over and refine in modules. The idea is to implement a dynamic resource request for each task based on the input fastq files. Most workflows already run FASTQC, so grab its existing metrics (total sequences, processed file size, read length/complexity) in a tiny “sizer” task, stick them into
meta
, and let downstream modules decide resources on the fly—e.g. `cpus { meta.reads < 1e7 ? 2 : 8 } memory { meta.reads < 1e7 ? 6.GB : 32.GB } time { meta.reads < 1e7 ? 4.h : 24.h } errorStrategy { task.attempt < 2 ? 'retry' : 'finish' }`—so small libraries finish fast, big ones get what they need, retries stay minimal, and we leverage data we already collect without extra commands. or resource request could use a look up table.
❤️ 1
p
This has come up a few times before. Agree that it sounds very attractive, the problem is that calculation of things like read length etc. requires reading the data.
If running on a compute system with remote / object storage then it also requires staging (downloading the data)
So even if the processing itself to get those metrics is fairly lightweight and quick, it still requires a lot of data transfer and time (eg. additional job queues etc). At which point it gets more difficult to justify whether the improved optimisation of downstream tasks is worth it.
An alternative approach is to run the pipeline more than once (eg. repeat runs, or a trial run with one or two representative samples). Then the metrics of that run can be generalised and built into an optimised Nextflow config. That's what the Seqera Pipeline resource optimisation feature does.
When that was first released we were excited that we could use it to optimise all nf-core pipeline configs, but in practice the workloads that different people use on the same pipelines seems to vary a lot, so it's difficult to generalise and hardcode. Leaving either to do it dynamically (your suggestion) or per-user retrospectively (the above link).
s
This has come up a few times before. Agree that it sounds very attractive, the problem is that calculation of things like read length etc. requires reading the data.
Yes, hence the suggestion to extract them via process that may well be already doing. Alternatively, can we get the size of the input files (in Gb)?
p
uff facepalm
Sorry, apparently I can't read
m
Thanks for the insights, Yes, I agree we don't wanna run extra, or transfer multiple times. In phase one, modules could take a switch key, based on the pipeline they are used in. If something like fastqc is on the workflow, then we already have precious data already extracted and we could use that to optimize resource request.
p
Right, then yes I think this makes a lot of sense 👍🏻
My preference would probably not to have any additional tasks, instead just have a bit of logic within the main FastQC module and some new outputs
Could then implement the same thing across a bunch of other modules, eg. fastp or whatever else
m
exactly, from my perspective most pipelines run something on the raw input, as you said with a bit of logic, this could be something to use the information for and don't throw something valuable away along the way. looking forward to see this in near future
s
well, changing the meta map is something that we've said shouldn't really be done (although I can't find it in the guidelines, so maybe we just start doing it until @James Fellows Yates jumps up and down 😉 )
Updating the meta map means that joins will (potientally silently) fail downstream if you aren't careful
j
(in a meeting atm, so not read this thread yet)
The example in first message can be done all in a
modules.conf
no?
Don't hjave to embed anything within the module?
s
James, surely you should be paying 100% of your attention to the meeting? I'm shocked, shocked! that you might not be doing so. I can't see anything on that page that says we can't update the meta map inside the module to add extra fields, although it feels like it is current policy.
j
Modules MUST NOT use ‘custom’ hardcoded meta fields
Unless I'm misunderstanding?
s
That just replies to custom input fields, not output ones?
j
Mayhbe should say 'must not use or refer to'?
Eh?
It has nothing to do with inpout or outputs?
Or you mean it's vague what 'use' could be interpreted as?
IT's not in an input/output section
s
Yeah, that is true. It feels a bit vague
j
Can update that sure
s
Although it'd be really nice to add dynamic allocation of the resources into the modules based on input file size as was being discussed, this is a big change in how some things are set, and adds more config complexity
j
Agree it would be good to add! But don't think hardcoded 'custom' meta fields in a module is required
m
cpus { meta.reads < 1e7 ? 2 : 8 } memory { meta.reads < 1e7 ? 6.GB : 32.GB } time { meta.reads < 1e7 ? 4.h : 24.h } errorStrategy { task.attempt < 2 ? 'retry' : 'finish' }
Finding the right formula isn’t always trivial. Even for read processors, resource usage may depend on the read length (especially relevant if you want to support all flavours of Illumina and ONT and PacBio etc), (in)directly on the species’ genome size, number of chromosomes, etc if the reads are mapped. Some tools do clever method for like sorting/grouping/aligning reads, which makes their resource usage in fact dependent on the polymorphism rate (which is relevant to support multiple organisms). The
time
constraint depends on the CPUs you’ve got access to, which can be multiple times faster across generations, on the disk I/O rate which can also vary greatly if you think local disk vs network disk, etc. It’s a great idea, and we try to do it in our pipelines, but one needs to be very careful about extrapolating the rules and having something that’s good for everyone/enough people and suitable for inclusion in nf-core/modules.
party this 1
m
I understand your points, I believe this is needed along the way. Does not make sense to request 6cpus and 2hours for both a small and large input. And about those retries, who's gonna pay for that! One option is to shift towards putting the burden on pipeline author who should maintain a tiny wrapper logic and set sensible thresholds. In this way, the author that wires the modules together can attach extra values to tuple it passes downstream. The next process just plugs them in. this could be something like:
Copy code
process FASTQC {
    input:  tuple val(id), path(fq)
    output: tuple val(id), path(fq), val(reads)
}

process TRIM {
    input:  tuple val(id), path(fq), val(reads)
    cpus    { reads < 1e7 ? 2 : 6 }
    output: tuple val(id), path("trimmed.fq.gz"), val(reads)
}

process MAP {
    input:  tuple val(id), path(trimmed), val(reads)
    memory  { reads < 1e7 ? 8.GB : 32.GB }
    ...
}
j
I agree with @Matthieu Muffato that the different inputs all influencing cpu/memory/time makes it very difficult to have a common 'linear' system that should be embedded in the modules themselves. I think it's more reasonable to assume a pipeline developer can gauge better for the typical types of uses/data for their pipeline, so I would opt for this like you say @Mehrdad Jaberi - that the burden should go on the pipeline author via the
modules.config
file. Furthermore, by having it in a config file that will be a way of setting such things that is more familiar for users who would have to more likely make a custom config themselves toadapt to their data. So for me: 1. No to embedding in the modules 2. Yes for finding ways to facilitate such a concept in as a user friendly system as possible that both users/pipeline devs can configure themselves :)
party this 1
p
But we have to get the numbers (filesize, num reads, whatever) from the modules. So step 1 is just emitting those as outputs so that they can be used downstream for whatever.
j
Ahhhhh now I see
Can those just not be taken as straight e.g.
env
outputs and then pulled into a meta in the pipeline?
p
Not sure about the
env
bits but yeah..
j
Or however
a
If I followed this correctly there are two separate but related points: • making some of the modules emit some of their usable stats in a different format • Using those stats for optimised resource requests I think the former is sensible and low-impact to do, I think the latter - as highlighted by James and Matthieu - is not generalisable enough to be useful implemented wholesale and is better documented in an examples/advanced usage section in the readme or some such, and used by developers running the pipelines as they see fit
nod 1
👍🏻 1
👍 1