Interesting findings: https://x.com/KLieret/status/2059643437964497161. I wonder if this benchmark is worth adding to OpenHands-Index/benchmarks repository.

Yeah, good point! We’d have to add it to the benchmarks repo first I guess

Yeah I was reading their codebase and it seems to be Harbor compatible (not sure how easy it is to port to OpenHands though). But each task seems to have a Dockerfile so it may simply require installing openhands on top of it the way we do it for SWE-Bench. Given that the performance varies a lot across several frontier LLMs, they seem to have found some high quality tasks that expose these differences (the number of tasks is somewhat low though). For e.g. the gap between kimi-k2.6 and gpt-5.4 is much larger here as compared to SWE-Bench Verified so this may be useful information we want to incorporate into the index.

OpenHands SDK runs in harbor for terminalbench, so I think it shouldn’t be too hard!