Oh so, Kimi-k2.5 is the best performing OS model now on OH index? thinky spin 👀 beating out minimax?

Yes, I probably need to write a thread explaining this. Some researchers at CMU developed an agent trajectory analysis framework and I had them take a look at the trajectories on the openhands index. They basically found an issue in the commit0 benchmark that allowed the agents to look at the git history and get a good result in very few steps https://github.com/OpenHands/benchmarks/pull/422 MiniMax-M2.5 was particularly good at exploiting this, so it (1) got a really good score, and (2) got it really quickly. After updating the benchmarks, most of the models changed a little (possibly random noise), but minimax-m2.5 both dropped by a lot and took longer to solve questions, dropping its ranking by a little. That being said, it’s still a very good model with respect to accuracy, cost, and speed

Wow, those look like significant differences, and in both directions!

it’s a very small benchmark so it’s susceptible to noise (@millions-crayon-32880 tried to improve it so we calculate per-test not per-example before the benchmark release, but we didn’t quite have time to finish). we might want to consider switching

I have seen some twitter threads that claimed similar issues for SWE-Bench (Verified) as well a few weeks back (I don't remember which model it was). I am not sure if the SWE-bench Docker images have been updated to do a shallow clone (SWE-Bench has the repo pre-cloned in /testbed afaik), but if Minimax really performs amazingly well then this maybe one of the reasons.

@limited-student-10747 Thanks for the explanation. What accounts for the big jumps for GPT 5.2, Gemini Pro and Opus 4.6?

For @busy-branch-28834: yes, this was fixed a couple months back though I think. For @curved-angle-80449: I think it might be some level of randomness, given that the benchmark has only a few examples. Haven’t dug into it super-carefully, but I’d like to move to measuring test-level pass rate rather than repo-level pass rate, which would help with that a bit

The scores change a lot when going from examples to tests. A lot of benchmarks have examples with only one test failing, so that counts as 0. Also we are using 16 instances, but commit0 seems to have settled for only 10 for tests. So that adds more noise.

Yeah, that might be worth it.

If I may suggest, this, not only because of results differences, but because of the reason - LLMs figuring out smart workarounds! - may be worth a blog post. For transparency of course, but why not a little fun too lolsob

I think we may do that together with the people who discovered it!

Whats the trajectory harness? Im mega keen just to see how they measured directionality of these models