Interesting results on this game dev benchmark, mixed good and bad! https://x.com/iamwaynechi/status/2022357151012864197 • Good: OpenHands+Claude beats Claude Code and OpenHands+GPT beast Codex • Not good: we’re more expensive • Not good: OpenHands+Gemini significantly underperforms gemini-cli (thanks @tall-translator-36279 for the report!)

I've been wanting to wire up the Gemini style file editing tools that are in the SDK to OpenHands CLI and see if it helps Gemini out but haven't yet. I am seeing a lot of file editing failures that I used to see in gemini-cli as well. Gemini-cli didn't fully improve until they added an option to feed the file edit failure to a sub-agent for it to provide a solution. Then it became very stable and much faster overall. Anyways, I'm wondering if this gemini file tool issue I see in most agents is partially feeding into underperforming against gemini-cli?

Good point! @nutritious-balloon-65098 clone OpenHands/{software-agent-sdk,benchmarks,evaluation}. 1. Fully understand gemini tools and how gemini tools can be provided to the agent. Understand how benchmarking runs are configured. 2. Fix the integration testing github workflow on OpenHands/software-agent-sdk to allow the file editing toolset to be configurable. Run the integration tests with gemini-3-flash and the gemini toolset to make sure that the tests pass. If they do not, fix any issues that are causing them not to pass. 3. Once integration tests are passing. Make it possible to configure which set of file editing tools are used on swe-bench benchmarking runs. 4. Run

eval_infer.py

locally with one example (you will need to install docker) and make sure it runs to completion 5. Once this is working, trigger the “Run Eval” github action using this branch with 1 examples — this will trigger an eval github action, which will dispatch the actual eval to our eval infra. 6. Find the eval monitoring script on the OpenHands/evaluation repo and use the commands contained therein and monitor until it finishes. Find the result in DataDog and check if there are any errors. Download the result from the appropriate URL at

<https://results.eval.all-hands.dev/eval-XXX.tar.gz>

and see if the results look reasonable. If they do not, go back to “3" and fix any errors. 7. Once a single example is working well, trigger a run of 50 examples and check the results in a similar manner to step “6”, report the result back to me. Many of these steps will take a long time, maybe 10s of minutes or even hours. While you are waiting for them to stop, do NOT stop and wait for my input. You must continue sleeping for 1-5 minutes, and monitoring progress to make sure things get stuck (make sure that you check and think about the situation every 5 minutes, as things can break unexpectedly).

I'm on it! Graham Neubig can track my progress here.