James Fellows Yates
04/21/2025, 2:47 PMFlorian Wuennemann
04/21/2025, 2:50 PMFlorian Wuennemann
04/25/2025, 5:22 AMpipeline_revision_optimized.config . If the revision is none, it means the pipeline was run from a commit branch. Unfortunately I didn't have time to compare how much of a difference there is between the optimized and the configured profile.
My suggestion would be that we include a config like this for each pipeline's test_full profile to reduce the amount of resources requests to what's actually use. This will save resources and CO2 emissions for each test_full someone runs in the future.
Let me know what you think and we can refine the strategy going forward. This was just a quick generation of optimized profiles based on previous performance on AWS Megatests!James Fellows Yates
04/25/2025, 7:14 AMJames Fellows Yates
04/25/2025, 7:15 AMFlorian Wuennemann
04/25/2025, 3:12 PMtest_full without having to modify the pipelines themselves. Can you share your idea here Max, please?Maxime Garcia
04/25/2025, 3:16 PMaws_tower profile that I've been trying to remove.
We could kill for real this aws_tower and create a new test_full config for specific pipelines, that would be automatically loaded whenever they're updated.
That'd be zero maintainance at the pipeline level.James Fellows Yates
04/25/2025, 3:17 PMMaxime Garcia
04/25/2025, 3:19 PMMaxime Garcia
04/25/2025, 3:20 PMMaxime Garcia
04/25/2025, 3:20 PMFlorian Wuennemann
04/25/2025, 3:21 PMRike Hanssen
04/25/2025, 3:21 PMFlorian Wuennemann
04/25/2025, 3:23 PMtest_full pipeline profile to config repo, that would force the optimized config on all nf-core test_full runs, potentially without people being aware of it. Positives: Save compute and CO2 emissions. Negatives, when it breaks its frustrating and can be confusing and hidden-ish.
• add optimized pipeline profile to config repo, that would need to be specifically activated when running test_full. Is more verbose but many people likely wouldn't know about it and use and hence, less CO2 savings and more dead polar bears sadpandaMaxime Garcia
04/25/2025, 3:25 PMMaxime Garcia
04/25/2025, 3:28 PMJames Fellows Yates
04/25/2025, 7:14 PMFlorian Wuennemann
04/25/2025, 11:13 PMJames Fellows Yates
04/26/2025, 5:04 AMJames Fellows Yates
04/26/2025, 5:05 AMFlorian Wuennemann
04/26/2025, 10:58 AMRike Hanssen
04/28/2025, 6:39 AMJames Fellows Yates
04/28/2025, 6:40 AMJames Fellows Yates
04/28/2025, 6:40 AMJonathan Manning
04/30/2025, 12:54 PMtest and test_full) so it gets as wide usage as possible. Updates might be a bitch, and we might be slow in adding optimised rules for new processes etc, but I think we can live with that.James Fellows Yates
04/30/2025, 12:54 PMJames Fellows Yates
04/30/2025, 12:55 PMJonathan Manning
04/30/2025, 12:56 PMJames Fellows Yates
04/30/2025, 12:56 PMJames Fellows Yates
04/30/2025, 12:56 PMtest_full is more popular outside of the megatests than I think...)Jonathan Manning
04/30/2025, 12:58 PMJames Fellows Yates
04/30/2025, 12:58 PMJames Fellows Yates
04/30/2025, 12:59 PMRike Hanssen
04/30/2025, 1:00 PMJonathan Manning
04/30/2025, 1:00 PMJames Fellows Yates
04/30/2025, 1:01 PMJonathan Manning
04/30/2025, 1:02 PMMaxime Garcia
04/30/2025, 1:30 PMJames Fellows Yates
04/30/2025, 1:30 PMJames Fellows Yates
04/30/2025, 1:30 PMJames Fellows Yates
04/30/2025, 1:30 PMEdmund
04/30/2025, 1:34 PMMaxime Garcia
04/30/2025, 1:35 PMEdmund
04/30/2025, 1:37 PMFlorian Wuennemann
04/30/2025, 3:19 PMtest_full profile to a fork of nf-core (https://github.com/FloWuenne/configs/tree/test_full_optimized_silent). When you then launch nf-core/rnaseq, the test_full profile from the nf-core/configs fork will be automatically applied in the background if you specify the following in your config or on the command line:
custom_config_version = 'test_full_optimized_silent'
custom_config_base = "<https://raw.githubusercontent.com/FloWuenne/configs/${params.custom_config_version}>"
2. Add test_full_optimized profile to a fork of nf-core (https://github.com/FloWuenne/configs/tree/test_full_optimized_extra_profile). Same principle, you can then specify test_full,test_full_optimized and this would apply the optimized profile. This is more verbose but would likely not be known / applied by many people.
I personally prefer approach 1 and do think that this is an easier to implement / less maintenance approach for pipeline maintainers. I think that modifying the resource requirements for tests where we specifically know how much they will use should be non-controversial though and if we tune the configs well for most deployments, I don't really see an issue with doing it this way.Maxime Garcia
04/30/2025, 3:24 PMtest_fullFlorian Wuennemann
04/30/2025, 3:25 PMtest_full goes Super Sayan