Hey, I just checked that evaluating GAIA with the OpenHands agent always had 2 instances with ERROR. At first I thought it was a problem with OpenHands/benchmarks, but after investigation it turned out to be a problem with the SDK, specifically with reasoning. You can find all the details in the issue: https://github.com/OpenHands/software-agent-sdk/issues/2482 Some proposed solutions are also listed, but I'm not sure which is the best one as I'm not very familiar with this part of the codebase. If someone with more experience in this part of the codebase could guide me on which is the best solution, I can open a PR with the fix. Thanks

I have run several evals on GAIA with OpenHands old version for one of my earlier papers - the easiest trick here is to have a hard-coded user message for evals that gets triggered when agent emits a MessageEvent, instructs the agent to not ask clarifying questions/submit answer using finish tool/or have some other domain-specific message based on observed error behaviours. Since the reasoning content for the current step is not included in the prompt for the next step, it may not be worth spending compute on steps without tool-calls since the subsequent actions have no idea what the earlier reasoning was and the LLM only sees empty message events.

Since the reasoning content for the current step is not included in the prompt for the next step

I'm not sure this is still the case

Oh wow - but I assume with the best models like Sonnet-4.6 or GPT-5.4 it is still the old logic and the scores of these models are what we care more about? My knowledge may have been outdated

Opus 4.5 requires the thinking blocks, but it looks like they might have removed that requirement for Opus 4.6.

Gemini 3.0 Pro from I recall, and I'm not sure if the requirement softened with 3.1 OpenAI reasoning models on

/responses

API also need something like, send back the

reasoning_item

as received I don't master anymore the intricacies of the Claudes, specially with these extended thinking, thinking blocks, dozen of details

(Specifically, they made Opus 4.6 only support adaptive thinking, which has this paragraph in the docs)

Thanks both! This is interesting (I personally used to dislike the idea of deleting the reasoning traces from prior turns as it is quite non-intuitive that the LLM can infer a workflow from only past tool-calls and no context about why these actions were chosen)

Makes sense to keep them around in the normal course of a conversation for that reason. The Opus 4.5 changes were giving us all kinds of problems with the context condensation though -- the combination of long agent turns and cryptographically signed reasoning blocks meant we had to start treating huge swaths of the conversation as atomic.

agreed - the older models like Sonnet-4 and GPT-4o used to have very concise reasoning traces in the

content

field that would reduce/remove this problem but the current reasoning traces have so many

Oh wait!

blocks lolsob

left a comment/possible solution regarding a similar problem I had with the preflight check. and a question regarding versions.