Hi all, The question about dynamic optimisation has come up at sanger again, my mentee's made good headway with the whole thing. We did however run out of time. Would there be interest in a interest_group meeting to talk about everyones approaches so far and how we can go ahead in the future? If so, when would be best for people? Should I start a poll? Thanks all.

I hope all the code will be publicly available with docs so, that it can be picked up later

Of course, i'll be compiling everything at some point this week.

I'm interested, but I've been focused on other things, so my stuff is just ideas

Would love to join to see what everyone is trying

@Damon-Lee Pointon Just wanted to check here to see if there was anything from your implementation available to check out online?

Hi @Florian Wuennemann as well as everyone in the thread. Apologies for not getting on this, it's been a year personally. Anyway. Yes everything is on GitHub across a few repo's. However, it is out of date... by a bit of a margin at this point so don't expect anything to work as hoped. I have recently made the argument to reopen this as an ongoing project however, so it will get updated. The original project is here: https://github.com/DLBPointon/SummaryStats This is a python tool which handled specific output from our pipelines (at the time this required custom output files and inclusion of custom groovy methods to get stats on input files). example here: https://github.com/DLBPointon/SummaryStats/blob/version2/test-data/1-1-0-runs/TreeVal_run_KPenguin2_1_FULL_2024-04-16_11-30-20.txt This resulted in graphs shown here: https://github.com/DLBPointon/SummaryStats/tree/version2/efficiency_graphs as well as: https://github.com/DLBPointon/SummaryStats/tree/version2/output-StatGraphsV2 Me and @Matthieu Muffato spoke about our approaches here: --- This evolved into V2 which was intended to replace the above, and we (@Hannah and @Lucas Aleixo Leal Pedroza where my mentee's at the time for this) came up with a system using a plugin to collect stats on run metrics and input file metrics, which could output a txt and/or submit via api to a DB. At which point an ML would create and optimised set of params. The idea would then have been that the plugin or a wrapper could be used to call the DB get the new optimised resource parameters. DB would have to be Team/Group/Organisation scale. Running a global DB would be insanity in my opinion and would require quite a bit of resource. The original SummaryStats would then be parted out and integrated into the plugin, DB and re-written as a standalone program. We were thinking of potentially re-writing in Rust too, Just the move from pandas to polars saved a HUGE amount of time and computer stress, however, understandably I can imagine the community would like as close to Python as possible. We got a long way, but ultimately ran out of time and this was 2 major Plugin system updates ago now. Alot has changed and our approach should too. Hanh's Plugin work: https://github.com/sainsachiko/treeval-summary-plugin Lucas' ML work: https://github.com/sanger-tol/summary-stats-analyses --- Now that I have brain space for it, it would be good to have a community chat to see approaches and where we should go. Especially as I have the green light to work on it, and it will improve our own pipelines substantially. Question is whether the Channel would be interested, a community Zoom chat early next year so people can get their ducks in a row for some presentations? I already run BGA which has a Youtube channel, so we can post there (public learning and presentations) if people want a record. <!channel> Meeting in Feburary time maybe?