https://nf-co.re logo
Join Slack
Powered by
I really liked the hackathon idea (Florian's team ...
# team-maintainers
e
I really liked the hackathon idea (Florian's team I think) of optimising resource requirements for nf-core pipeline test profiles. I guess the most important one is the test_full profile since it runs for every release on aws, but I think we should have optimised mem/cpu/time requests written down for every test configuration / module anyway, since the usage is expected. Just to make sure, the best place for that would be in each test*.config, inside a process {} block, right?
f
Thanks @Evangelos Karatzas! I will finish up our work from the hackathon and post the results in #C06MD9L16UF soon (maybe on this flight eyebrows 🫠 ). The idea is to basically add a configuration to test and 
test_full
that rightsizes them better to the data, since that will never change. This should ultimately reduce the overall provisioning of cpu and memory for pipeline runs, leading to less CO2 and less money wasted.
❤️ 2
During the hackathon we basically tried to run all test for all pipelines and 
test_full
for pipelines that had an AWS_Megatest in the last 2 years (so pipelines with a release). That way, we could get optimized configs for them and make a PR with those configs to the 
dev
branch of each pipeline. Longer term, I think some automation might be nice, that will do a diff and see if the config is already present and only make a PR if there are changes / new processes.
e
I do that in my non nf-core pipelines, but it's high time this gets widely adopted for all nf-core ones 😄 By config, I guess you mean the process {} block in these two config files. I also think that for some of the modules, the label configs might already be optimal, so an override with 
withName
to give the same resources might be an overkill. So it might be tricky to know if a new process isn't there because it's already optimal through its label, or because it was forgotten.
f
Very true. I guess we can see what works best but since test data usually never changes during pipeline dev and lifetime, I don't see a problem with simply having a fully optimized config with most process names in there. Again, this is only for 
test
and 
test_full
(and other pipelines could consider doing it for their other tests) to reduce wasted compute. People will still need to figure out optimization for their "real" data.
e
And I guess for the real data could go into the base.config, or someplace else that we decide on
f
I mean, for real data I wouldn't provide an opinionated resource config as part of the pipeline to be honest, as this will change a lot from user to user. The user should generate this and provide at pipeline launch IMHO.
e
Agreed
Ok, something weird that I've never paid attention to before; running nf-core/proteinfamilies test profile on a local machine seems to require different mem resources for the same module than when running it on slurm. Local:
Copy code
rss 5.8 MB Real memory (resident set) size of the task
peakRss 5.8 MB Peak of real memory
vmem 9.5 MB Virtual memory size of the task
peakVmem 9.5 MB Peak of virtual memory used by the task
Slurm:
Copy code
rss 2.6 MB Real memory (resident set) size of the task
peakRss 2.6 MB Peak of real memory
vmem 4.3 MB Virtual memory size of the task
peakVmem 4.3 MB Peak of virtual memory used by the task
Is this normal? If so, based on which execution should I set my mem configuration? Maximum I guess? peakRss or peakVmem?
f
So there can be a lot of reasons for this. Docker versus singularity, different JVM settings, memory access patterns on local vs HPC, the list goes on. I would set the config to where you would most often deploy the pipeline or then one with higher requirements, depending on your choice!
e
Thanks Florian! For now I would not touch the cpus and stay with the option defined by the label. Because I feel the only way is too deeply research each tool's best settings (mostly the hard cap that the performance increases). And for time, I am overriding, close to 2-5' each
p
Also note that 
memory
as per the process is typically used to set the cgroups memory limit which includes quite a bit more than what 
time
(or wherever that output is from) for one process will show.
e
Hmm didn't get this..is there a link with a deeper explanation around this?
p
Do you mean the nextflow angle (l think not) or cgroups (loads, but I'm not sure which you would feel most accessible).
e
Any documentation offering per-process 
memory
knowledge, further than the expected. I observe that even in the same system the max memory for the same process might differ between executions. And for local executions (my linux laptop), it isn't even killing the jobs if they end up using more memory than I've defined in the respective 
withName
, which I find weird
p
If you're using the 
local
executor and don't run with docker, there's nothing to enforce the limit set.
e
I do use singularity
p
It doesn't enforce (it's so commonly used with a scheduler which will enforce it for the job).
e
Alright, this makes sense. The local executor is probably just a default value, without a scheduler to enforce..thanks!
p
Not default, just no enforcement.
e
Ok, so ofc, the memory settings I tested on slurm and locally were not sufficient for the AWS release full test in the end.. https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/1PZQNxD5SQBuNC/v2/logs 1. Is there any hacky fix to make this release's tests pass somehow? (e.g., bymp retries to 3x or something) 2. Should I remove the hand-picked memory assignment from 
test_full
profile in the next release? What would you recommend @Florian Wuennemann?
Open in Slack
PreviousNext
Powered by