Hi all! I’m trying to replicate the gpt-5-mini scores here from the OpenHands Benchmark Performance google sheet that reports 62.6% on SWE-Bench verified, but I’m consistently getting a score around 55% (e.g., 266/476 resolved). Even if I account the failed cases to be all resolved, that would bring it up only to 58% (290/500). I’ve made sure that reasoning was set to high. Are there any other configs that I should be tuning to match the reported score?

Hi Justin! I don't know if you are experiencing this, but I noticed something similar with a run of Claude 4.5 Sonnet where I had ~10% lower performance than the reported OpenHands score. It turned out to be due to an evaluation bug with the sphinx and astropy repos, where there was a difference between the

/testbed

and

base_commit

repos causing the applied patch to be reverted to the base state (and so all runs failed). Other than that I had slight differences in the system prompt and ran the model for max 100 steps instead of max 500, but the final score was 70.80%, compared to the reported 74.6*%.* Here are links to the Github issues/PRs for the repo issue in case they are helpful (I applied the change from the second one to resolve my problem): https://github.com/SWE-bench/SWE-bench/issues/228 https://github.com/Kipok/SWE-bench/pull/3

Thanks a lot Nicholas! yes I’m seeing all sphinx cases fail, so that sounds relevant to me too, I’ll check out the links you shared, much appeciated 🙂

So I think for Claude 4.5 Sonnet the setup may be slightly different? It's actually reported under the v1 SWE-Bench Verified tab (not the same as gpt-5 mini), I think it's for a single try but uses up to 3 attempts to obtain a patch (e.g., if 1st and 2nd attempt return an empty patch). I was sent this link where you can download trajectories for the Sonnet runs: https://github.com/OpenHands/openhands-index-results It seems to be set to max 500 steps. As I said though, this is for a different set of models on the sheet so the setup may be different from gpt-5 mini. Hope this helps!

great, thanks 🙂