Hey all, especially @strong-ice-84145, thanks for all of the work on this! Are we basically in a good place with evals right now? Or do they still die for some reason?

Eval runs don't/shouldn't die anymore: • async should shield us from OOM • image building timeout was set to 24h which is much bigger than the current 10 hours it takes for swebench/swtbench However the image builds still take much more time than a month ago (=5hours) The previous hacks allow us to win 10% build time but it's not that much Still trying a few hacks here • https://github.com/OpenHands/benchmarks/issues/538 • https://github.com/OpenHands/benchmarks/issues/537 • https://github.com/OpenHands/benchmarks/issues/535

OK, thanks a lot. I know you’ve been working super-hard on this, but maybe it’d be worth getting 1-2 other people to help resolve this issue once and for all. If you had some more help what would be the most helpful?

Yes definitely!

Thanks! What parts do you think we could carve off and have someone look at? Or do you think just having a dedicated code reviewer would be enough?