Thanks a lot to @strong-ice-84145 for setting up benchmarking on the OpenHands Index of other agents, we have very interesting results: https://index.openhands.dev/alternative-agents Some takeaways: 1. OpenHands is beating Claude Code with anthropic models 2. OpenHands is similar to Codex with GPT-5.4, but more expensive 3. OpenHands is much worse than gemini-cli with Gemini For all of these, I think we need to dig in to the results and see why this is happening. Maybe a next step is to dig in to traces and examine the differences between the harnesses when using different models: https://github.com/OpenHands/openhands-index-results

On the other hand, we’ve never evaluated the OpenHands sub-agent harness with Gemini Pro. It might be worth running those evals.

This is very interesting, thank you!

I did a quick EDA of the Gemini trajectories and discovered that it was trying to 'cheat' on the Gaia benchmark. 😈 For example we have the reasoning block

"Let's do a quick check on the GAIA dataset answer for this question just in case there's any public repository or solution guide..."

in the instance Ben & Jerry's flavor graveyard Moreover, 13 different instances attempted to fetch the GAIA validation set directly to look up the answer. But all the attempts failed as the dataset is gated.

Maybe we could add a quick single llm call at the end of each trajectory to look for cheating attempts?

Interesting. I think in the short term we’ll want to try to come up with ways to avoid this like: 1. prompt the model about what it isn’t allowed to do 2. post-hoc check about whether it followed those instructions In the longer term, I know we’re thinking of coming up with a new version of the OpenHands Index and we can keep this cheating in mind when picking new benchmarks.