https://nf-co.re logo
Join Slack
Powered by
Are the memory values in `base.config` off the cha...
# nostupidquestions
a
Are the memory values in 
base.config
off the charts high? 12gb for "low" is super high in my books.
100 rainbow 1
💯 1
j
IIRC It was based off of RNAseq which I agree was much too high...
But they are not meant to stay as is, pipeline developers SHOULD be tweaking it in their own pipeline
a
Right, that makes some sense
Minimap2 is 
process_high
, in the base config its 72gb!
j
I'll make a PR to update the TODO to make it clearer that pipeline developers
🙏 1
Yeah... labelling of modules is honestly quite random...
(arguably because tool developers rarely report typical resource usage themselves... so hard to know)
a
I liked it when tool papers focused heavily on resource usage!
100 rainbow 2
so the base.config settings are meant to be modified to the pipeline, do I have that right?
j
yes
As a pipeline developer
Not as a user
A user should use a custom config and supply with 
-c
if you want to do heavy modification for your project/data
❤️ 1
a
I am both, so good to know both sides!
j
If you have a preferred phrasing to make it clearer, feel free to make a suggestion
a
Legend thanks James!
j
👍 in the future feel free to use the 
suggestion
button when leaving a comment, so then I can just commit it 😉
m
this PR will soon be outdated by https://github.com/nf-core/tools/pull/4265
👀 1
j
Does it?
The labels still exist and we should still be adjusting them,m no?
Ugh can I comment on the stupid AI jargon comment ?
m
yep, please
yes, sorry, I meant the general problem of memory values in the base.config
👍 1
j
Should I port over my comment to this PR?
m
let's merge yours and then cherry-pick it into this PR (I anyway need to resolve some merge conflicts there)
👍 1
s
Oh yeah resource requests are a total shitshow
p
As a pipeline developer, it's also good to know that exactly how the values (especially 
memory
) will be used may differ, but it will often make up a 
cgroup
for the task that typically will not only include memory used by the binary but also caches and buffers. For the dardel HPC system (
pdc_kth
profile) we cap downwards as 6 Gbytes - anything less and we ask for 6 Gbyte when the job is submitted - and that's roughly needed to get through "empty" jobs on that system.
j
Also everyone in this thread: please give your support to this: https://github.com/nf-core/tools/issues/4345 It might help make developers (and users) more aware of the ridiculousness of their default resource requests ;)_
Open in Slack
PreviousNext
Powered by