Hi all! On the OpenHands site we just published a guest blog post SWE-fficiency: Evaluating How to Fix Code, Not Just What to Fix by @jolly-orange-24543! It's an interesting benchmark that uses OpenHands as a harness, and evaluates the ability of language models to write efficient code. Hope you enjoy it, and we'd be happy to have any discussion here.

Hi Graham sir and @jolly-orange-24543! Thanks for sharing this blog - this is very interesting line of work. I was reading this blog and Jeff's paper, and was wondering how hard is it to create a training counterpart of this dataset for RL/SFT experiments? It might be worth investigating since the best proprietary models perform poorly here, whereas the same models are great competitive coders (which often needs somewhat similar code optimization skills atleast at a high-level?). From what I understand, these are the two main main bottlenecks right now: 1. Identifying repositories that have such performance enhancement PRs (beyond those in test set): do you think there are enough repositories beyond what you cover that have this property? 2. Curating per-instance performance workloads manually: From Appendix C.4 and Appendix I it seems that you tried automating this step (which implements a script that measures code speed/performance) but found that it did not work that well. I am curious about the total wall-clock time you spent per instance (or overall) to get these workloads? (I want to estimate the effort involved in repeating this step manually on a larger scale 😅) [This is just a thought, I don't think it would be an easy task to train current OSS models to perform better at this task, especially not with RL, due to high reward sparsity and risks of reward hacking, though SFT from proprietary LLMs may still work/help]