<!channel> reminder of monthly maintainers team me...
# team-maintainers
j
<!channel> reminder of monthly maintainers team meeting this week on Friday 14:00 CEST (Berlin time) Please give a salute if you plan to attend!
🏖️ 3
🗓️ 1
salute 8
f
I am unfortunately out due to a Dr. appointment for my son on Friday. I have added one discussion point to the agenda about resource optimized config profiles. I hope I can add an overview of overutilization of resources for all AWS megatests by Friday as a showcase!
❤️ 1
I was able to put something together regarding the resource labels. https://github.com/FloWuenne/megatest-resource-optimization/tree/main/optimized_configs I used the API for Seqera Platform and the AWS Megatests workspace to fetch successful runs since January 2024 for all pipelines and then created configs based on actual resources used by each pipeline run. The configs are named
pipeline_revision_optimized.config
. If the revision is none, it means the pipeline was run from a commit branch. Unfortunately I didn't have time to compare how much of a difference there is between the optimized and the configured profile. My suggestion would be that we include a config like this for each pipeline's
test_full
profile to reduce the amount of resources requests to what's actually use. This will save resources and CO2 emissions for each
test_full
someone runs in the future. Let me know what you think and we can refine the strategy going forward. This was just a quick generation of optimized profiles based on previous performance on AWS Megatests!
j
I think this is a great idea!
I think the main question is how to update this atuomatically... for example for after the first release we can say please make an optimised config using these instructions, but then do we want to ask developers to update this on every release? But anyway - something for themeeting
f
@Maxime Garcia had a great / crazy idea of how we could easily apply these configurations to
test_full
without having to modify the pipelines themselves. Can you share your idea here Max, please?
m
yeah, so my idea, is that instead of adding such configs in the pipelines' repos, we hop on the configs repo. We have pipeline level infrastructure configs. We even have an old
aws_tower
profile that I've been trying to remove. We could kill for real this
aws_tower
and create a new
test_full
config for specific pipelines, that would be automatically loaded whenever they're updated. That'd be zero maintainance at the pipeline level.
j
tenor_gif4446293376037718326.gif
m
@Rike Hanssen come here
I created the pipeline level infrastructure level just for that
f
I will make a POC with one pipeline and a fork of configs next week and share here. Discussed also with @Rike Hanssen and we thought a good way to go would be to come up with a strategy of how we would like to implement the optimized configs here with maintainers and then present that idea to pipeline maintainers since they are the ones that have to deal with any issues that might arise.
r
ok. Repeating my thoughts: love the elegance. Downside: the config is hidden in an already complex infrastrcuture. I don't want to make things more complex for pipeline developers.
f
Basically there are two ways to go: • add
test_full
pipeline profile to config repo, that would force the optimized config on all nf-core
test_full
runs, potentially without people being aware of it. Positives: Save compute and CO2 emissions. Negatives, when it breaks its frustrating and can be confusing and hidden-ish. • add
optimized
pipeline profile to config repo, that would need to be specifically activated when running
test_full
. Is more verbose but many people likely wouldn't know about it and use and hence, less CO2 savings and more dead polar bears sadpanda
m
so you all agree with using the pipeline specific infrastructure level config?
➕ 2
So now it's either we hop along and use the pre-existing profile, or add a new one. I'm guessing if we add a new one, we could either pre-install it by default when running on the platform, or add it in the GHA as an extra profile
j
@Rike Hanssen what is your concern about it being hidden exactly? The optimised config is essentially for a singular 'private' purpose for nf-core, no?
f
I think what @Maxime Garcia idea entails is that if we add a test_full profile to the institutional pipeline profiles, then this effectively gets added to the existing test_full everytime someone runs test_full. This additional configuration for process resources would be not in the pipeline repo and therefore hidden as Rike said.
j
Got it - that doesn't really concern me too much though: 1. Like I said before, technically the profile is technically 'just for us' (even if others can use it). It's at your own warranty, as with the normal test profiles 2. The optimised config is essentially adapted to the data itself right? So other than maybe Walltimes - the RAM/,CPU requirements should apply everywhere the data is used as the data itself is not changing between each location (practically that is not actually the case ofc, because of hardware differences, but I don't think it makes a huge amount of difference)
We could also have in the config description a link to where those settings are stored for example, if someone REALLY wants to know
f
I agree with all of your points @James Fellows Yates and therefore think we should go with Max idea! 🚀 Again, will make a quick demo to show it works and if it does can make a larger PR to add configs for a number of pipelines. Down the line, could even add stuff to the Mega test CI/CD that gets the optimized config after a successful megatests!
🤩 2
r
> 1. Like I said before, technically the profile is technically 'just for us' (even if others can use it). It's at your own warranty, as with the normal test profiles I disagree with this. People are using the test_full profile to run pipelines on their cluster to see if all is good before actual data submission (at least we did this a lot at qbic, I could see others do this as well.) Also I don't think we should be that possessive. It's a good thing if people test the pipeline thoroughly before submitting on their data which either can get expensive without getting the desired results or just waste a ton of their time and hours to help debug cluster issues > practically that is not actually the case ofc, because of hardware differences, but I don't think it makes a huge amount of difference) yep exactly. at the very least the test_full config should have some generous wiggle room. Different schedulers also vary in how aggressive they are to kill things when something goes out of bounds. AWS is pretty aggressive so hopefully that gives us a good baseline. IMO we should still add a little on top. It's more expensive to rerun tasks with twice the memory amount instead of just adding another GB from the get go
👍 1
j
Agree with that 🙂
I think that's reasonable
j
I'm with the polar bears, chuck it in the regular profiles (
test
and
test_full
) so it gets as wide usage as possible. Updates might be a bitch, and we might be slow in adding optimised rules for new processes etc, but I think we can live with that.
j
What's wrong for supplying it via nf-core/configs?
We don't have to modify any code, and we can update it on the fly as necessary
j
Because it doesn't ship with the pipeline, which I would anticipate would mean it gets less use than if it was just baked into pipelines. But I guess there's no harm in it going into configs in the first instance, and then inviting pipeline maintainers to adopt.
j
Use from who?
(I'm curious if
test_full
is more popular outside of the megatests than I think...)
j
I really don't think of them as megatest-only. They're a way of anyone seeing how things work with realistic data. We use them like that internally all the time, and I imagine others do too. The minimal tests don't bring up e.g. infra/ network problems in the same way.
j
I see
But teh nf-cre/config are still used in those cases, no?
r
We used them a lot at QBIC too to test the cluster
j
It does make sense for us to generate them automatically based on megatest runs. But as a maintainer I want to take them and bake them into the config so that anyone using that profile uses no more resource than required. I'd apply the same to the minimal profile.
j
Fair enough. I guess I just feel many pipeline devs won't be bothered to update (themselves) on release the newly optimised config for every release. But I guess we can make tooling for that once we've tested the nf-core/configs versions are actually shown to work.
party this 2
j
I say we host centrally, pipeline folks can adopt or not
👍 1
m
So now the question is should we host centrally (on the configs repo), or should we host at the pipeline level. In both cases, we agree that we'll be super-charging the test_full profiles, right?
j
Yes, and actually I think we are going for both
😬
A semi stable one within the pipeline, and a 'live' one that can undergo optimisation between rleases
❤️ 1
e
m
both?
f
Ok, I had a chance to test out two POCs that we discussed (since adding the config directly to the pipeline of course will work, I didn't test this on a branch): 1. Add
test_full
profile to a fork of nf-core (https://github.com/FloWuenne/configs/tree/test_full_optimized_silent). When you then launch nf-core/rnaseq, the
test_full
profile from the nf-core/configs fork will be automatically applied in the background if you specify the following in your config or on the command line:
Copy code
custom_config_version      = 'test_full_optimized_silent'
    custom_config_base         = "<https://raw.githubusercontent.com/FloWuenne/configs/${params.custom_config_version}>"
2. Add
test_full_optimized
profile to a fork of nf-core (https://github.com/FloWuenne/configs/tree/test_full_optimized_extra_profile). Same principle, you can then specify
test_full,test_full_optimized
and this would apply the optimized profile. This is more verbose but would likely not be known / applied by many people. I personally prefer approach 1 and do think that this is an easier to implement / less maintenance approach for pipeline maintainers. I think that modifying the resource requirements for tests where we specifically know how much they will use should be non-controversial though and if we tune the configs well for most deployments, I don't really see an issue with doing it this way.
m
Yeah, definitively I think whatever we do, we just super-charge
test_full
this 2
🚀 2
f
profile
test_full
goes Super Sayan
✅ 1