I have been testing how much the coding harness changes the behavior of the same model. (Working on material for my upcoming harness engineering workshop.) I gave OpenHands, Pi, and OpenCode a longer full-stack project using GLM-5.2 in the same clean AWS environment. The differences were much larger than I expected, including a 2.5x difference in input tokens and more than a 3x difference in cost. I understand this is one data point, but still think there is plenty to learn. I would also love to see people run similar comparisons on their own real tasks and share what they find. More details in the thread.

One quick question: what was the cache hit rate?

The place I'm struggling with the conclusion is that it seems to imply that all three harnesses produced equally good solutions at the end. Am I confused?

Good questions. (I can give you a more detailed analysis document if you want) The provider-reported cache-read rates were: • Pi: 97.5% of input tokens • OpenCode: 92.7% • OpenHands: 72.3% OpenHands still received 4.89M cached tokens, but it also sent 1.87M fresh input tokens. Pi sent only 80K fresh input tokens and OpenCode sent 201K. Caching made the repeated context cheaper, but OpenHands was sending enough additional and changing context that its cost remained much higher. The three solutions were not equally good. OpenCode passed all eight verifier sections. Pi passed seven of eight because it created a duplicate browser marker, which broke part of the UI contract. OpenHands satisfied all eight sections after we corrected an assumption in the verifier that was not specified in the original task. Even 8/8 does not mean two solutions have equal overall quality. The verifier checked the required behavior, including persistence, concurrency, worker behavior, API, CLI, and browser contracts. We did not independently score maintainability, architecture, visual quality, or code clarity. The narrower conclusion is that OpenCode and OpenHands both met the tested contract, while OpenCode did so with substantially less time, context, and cost. Pi was the lightest in model calls but missed one specified UI behavior.

I like the idea. I used to run a lot of head to head tests with models. I don't still do it because I found that a more complex the prompt makes more erratic the results. In a shootout setting, a model might be the clear winner today and then totally suck on a different day, same prompt.

Very interesting. A few remarks: • We already compared OpenHands vs Codex vs Claude Code in our benchmarks via ACP. The tasks were popular benchmarks (Swe-Bench, Gaia, ...) We found that OpenHands was competitive with CC/Codex. index.openhands.dev/alternative-agents I find it weird that OpenHands/CC/Codex are on-pair on those benchmarks but OpenHands is worse than Pi/OpenCode on your single task. A few possibilities: ◦ we did not use the same metrics in our previous comparisons. (I think we did, though) ◦ or your single task is not representative of the benchmarks we used • It would be nice to scale your own experiment to a higher number of tasks. You can probably tweak a bit the code to use the benchmarks repo and compare those harnesses on swebench / commit0 / gaia? •

By the final call, OpenHands was sending 97,500 tokens, compared with 57,700 for OpenCode and 73,900 for Pi.

This could be irrelevant: for long tasks there might have been condensation.

I think there was no condensation, maybe Rajiv can confirm or deny According to the report, OpenHands used 95 iterations and 97k tokens. Condensation is by default at 240 events unless that changed recently - and 97k is not near context window 🤔

Yup, no condensation. This is also a greenfield task, which might be different than many other tasks folks run.Its different than the Greenfiled benchmarrk used in openhands index. But it seemed like a fair general task to use. I also think the GLM 5.2 is probably a lot more chatty. I ran this with Codex and GPT 5.5 and it finished in about the same amount of time, but a lot less tokens.

Was the prompt a full spec like a PRD or something or more abstract? Did the prompt tell the models to build MVP and stop? I'm sorry if I'm just getting lost in memories of my own mistakes but I can't help but to feel like the story is partial if the solutions aren't compared on merit and if the terms weren't sufficiently mapped. A trivial decision, such as whether to add a real database or store data in json and defer the decision would totally nuke token comparisons.

Going to put all this into a repo, so it will be easier for questions: Your concern is valid. The long-project prompts were detailed implementation contracts, not abstract “build something useful” prompts. They specified architecture, public APIs, persistence, concurrency behavior, CLI commands, and acceptance checks. For example: • Durable jobs explicitly required SQLite, atomic claims, retries, recovery, a specified API, and cross-process persistence. • Incident operations explicitly required SQLite, optimistic concurrency, leases, audit events, HTTP routes, CLI commands, and a browser interface. • The spread-plate application specified behavior and DOM contracts, but left substantially more visual and implementation freedom. The prompts did not say “build an MVP and stop.” They said to build the complete requested experience, run tests, exercise workflows, and inspect the final diff. That matters because an agent could continue polishing, write more tests, or implement additional abstractions after satisfying the core contract. We did compare functional merit, and the results were not presented as equally good: • Durable queue: Pi and OpenCode passed 10/10 hidden checks; OpenHands passed 8/10. • Incident system: OpenCode and OpenHands satisfied the eight contract sections; Pi scored 7/8 because of a duplicate browser marker. • Spread plate: all passed the automated behavior contract, but we did not conduct a sufficiently rigorous blind visual and maintainability comparison. So the safest interpretation is:

The token numbers measure what each harness spent attempting the same detailed contract. They do not, by themselves, establish that the resulting implementations were equally good.

Your database example was controlled for the persistence projects because SQLite was mandatory. But subtler choices were still open: schema design, transaction strategy, amount of testing, abstractions, UI polish, and whether to add extra functionality. Those choices can materially change token use.

This is fascinating! I've been approaching same or similar ideas from a different angle. I don't want to pollute or hijack your thread but I'll tell you all about it sometimes. rn I gotta dig into this! lol

Awesome! One reason for this thread is a bit more testing around this to understand the promise and limits of harness choices. It would be great to get more eyeballs and more people testing in diverse environments. (This is how I found out i had 150 tool calls in my local agent canvas, yikes!)

To note, this looks like a potential bug, something blew out the OpenHands cache

Yes—OpenHands ended at a 72.3% aggregate cache-read rate. But it wasn’t consistently getting 72%. The behavior was almost binary: • Most calls reused roughly 91% of the prompt. • 20 calls received zero cache reads, including the expected first cold call. • Near the end, 10 of the final 11 calls were complete misses, each resending about 93K–98K tokens. Those misses pulled the overall rate down sharply. Without the 19 unexpected zero-cache calls, the estimated cache rate would have been about 91.7%. We found no OpenHands action that clearly invalidated the cache: • No condensation occurred. • The conversation history grew normally. • The same 22 tools remained available. • The zero-hit streak happened across calls only seconds apart, so ordinary cache expiration is unlikely. The pattern is most consistent with a provider/proxy cache routing or availability problem, although the ledger doesn’t expose enough backend information to prove the exact cause. OpenHands amplified the damage because its uncondensed prompt had grown to nearly 100K tokens. Every provider cache miss therefore became extremely expensive. So: (Repo is up and adding the traces) OpenHands’ cache reuse became unreliable late in the run, but the current trace cannot tell us whether that was caused by OpenHands request construction or provider-side cache behavior. Its large, uncondensed context made the misses particularly costly.

At a glance, my agent reported:

Yeah, this cache is very suspicious.

I am going to run with a few changes and I will report back - going to more carefully analyze the cache

Actually I had an issue with GLM-5.2 previously as well, so it might be a GLM+openhands issue

Hmm, I know i have had a few issues with Agent Canvas and GLM 5.2 this last week and wasn't sure where the issues was, hmmm I will try with Sonnet as well.

If you wish to test for OpenHands prompt caching behavior , I would suggest another LLM different from Sonnet/Claude - all Anthropic has a different implementation

I reran our longer Incident Operations Center benchmark to see whether the original harness differences held up. First, I repeated native OpenHands with GLM-5.2 on the same clean AWS environment. Then I kept OpenHands fixed and switched the model to Sonnet 4.5. Both runs used fresh workspaces, no repair round, provider-reported token usage, and the same external eight-part verifier. The short version: the second GLM run was faster and cheaper, but it did not close the call or context gap with Pi and OpenCode. The GLM gap does not appear to come from provider problems. The second OpenHands run had no failed provider requests, faster responses, and much better cache use. It still made 129 model calls compared with 69 for Pi and 76 for OpenCode. It also sent much more context on each call, averaging about 79K input tokens compared with 46K for Pi and 36K for OpenCode. Those two effects compound. OpenHands calls the model more often, and each call carries more system instructions, tools, and conversation history. Provider health changed the time and cost, but it did not change the size of the agent loop. Full setup, results, and sanitized evidence: github.com/rajshah4/harness-benchmark/blob/…/incident-repeat-trials.md

What was the cache hit rate?

Thanks! Do you think OpenHands just did more work or it did the same amount of work but took more turns?

(This is what my agent is saying, i haven't reviewed this stuff myself) It looks more like OpenHands did roughly the same productive work over more turns, plus some additional self-verification. The second run made 129 model calls instead of 95, but produced only 6% more output and passed the same external contract. Input tokens increased 51% because every additional call carried a large and growing conversation history. So I would not interpret 10.2M input tokens as 51% more useful work. Most of the increase appears to come from a longer agent loop: smaller steps, more calls, and repeatedly processing the earlier context.

OK, I think that’s definitely worth investigating, esp. if you could share the trajectories.

I'm digging in and finding real noise. I'll share once I validate.

Thanks fro flagging!

Yea, I have tried sharing everything in the github - @modern-fireman-99755 is always asking for the trajectories 🙂 @straight-dog-12443 be careful measuring, its not easy, I went through many iterations: Agent interfaces, ACP adapters, and tracing tools did not always agree on input tokens, cache reads, or cost. Missing telemetry also cannot be treated as zero. Run some test cases to make sure its running fine first.

perfect, thank you 🙌

Yea, I have tried sharing everything in the github - @modern-fireman-99755 is always asking for the trajectories 🙂

Haha 🙂 I don't believe if I don't see the trajectories...

@average-processor-75195 could you turn off the web browser and try again?

Hmm. Yea. Good idea.

If you checkout main now, the agent should just have access to 11 selected skills and not the whole marketplace. I suppose this will also play a role

I think that might be the main source of difference

I haven't upgraded yet. It's all 1.15.0. I can checkout from main and run another test tonight.

Tell us if it’s still not competitive. Vasco did some other work on making the system prompt more configurable, so we could pare it down a bit too.

@average-processor-75195 Just please make sure that you are using only the 11 defaulted skills. To be comparable. (I suppose you have a lot of other personal skills)

I am running all this on a AWS instance, so its separate from my personal setup. I just go with the defaults out of hte box.

Validated. Short answer to Graham's question: more turns, and every turn carries the same fixed payload. Method: decompose each model call into system prompt, tool schemas, history, and latest message; three of Rajiv's short-suite tasks, default agent (19 tools), two non-Anthropic lanes (MiniMax-M3, gpt-4o-mini); all runs completed. The constant: ~9K tokens of preamble per call (3.3K system + 5.7K tool schemas). ~84% of a first call, ~79% of the average call on the rename task. On the 26-call refactor, re-sent tool schemas came to ~149K tokens, about 27% of the run's provider-reported input (second lane: ~20%). For the browser-off rerun: the browser set is 14 of the 19 tools but ~2.4K of the ~5.7K tool tokens, so expect the tool preamble to drop toward ~3.3K. Skills land in the system bucket, separately measurable. History overtakes tools around call 8-11, which fits the Sonnet table: same loop, half the turns (66 vs 129), so the fixed cost stops dominating. Counts are estimates checked against provider usage on every call: 0.97-1.00 with a mapped tokenizer, 0.85-0.97 on fallback. Missing fields kept unknown, not zero. Per-call data and the small Metrics change behind this are available if useful.

Still crunching the numbers, but going to 11 skills has made a big different with GLM 5-2 A lot less calls and reduced token usage

is that with browser on or off?

whatever the defaults are on @modern-fireman-99755 latest version

Sounds a bit surprising, unless we changed the browser being on by default?

In the previous trajectory, I think the browser tools were on or at least my agent told me that they were on in the source of a lot of the prompt bloat. Removing skills should help, but I don’t think it would help that much because we do progressive disclosure.

Exactly, part of the task required using a browser interface. So this led to the tools being loaded and filling up the prompts right from the start. (and I think the browser mostly failed)

Tools are always loaded, though. Maybe you can tell your agent to look at the system prompt in the last trajectory, for the list of tools, not guess from settings

It looks like the first time it ran, with GLM it leaned on browser use which blew up the prompts. Openhands with Sonnet, skipped using a browser and instead verified the UI by using curl + grep parsing. But that ended up missing one check 7/8. Now Opencode which also had less token usage than OpenHands on 5.2, skipped using a browser, but wrote better code (Partly by creating good tests) and got 8/8. So i suppose there could be a cases where the browser does make a difference, but in this case, as Opencode shows you can get away without a browser and rewards fewer tokens. When we took the browser away from OpenHands (latest version of Agent Canvas with a small set of tools), it used the same approach as Opencode and was able to pass the front end tests with 8/8. This also saved a lot of tokens. Makes my head spin, more data tomorrow

yeah, sounds good. and the results may be different every time. we probably shouldn’t read too much into n=1

I think there is something there to how we use the browser which is different than other ACP harnesses in Agent Canvas. It is eating a lot of tokens and probably not needed so much. There is also the other issue of number of tools in MCP, which didn't come up here since I removed all the MCPs for this test.

It’s definitely helpful if you actually click on UIs to do QA of user interfaces

If we have an edge on UI, we should probably promote or highlight that more. I see it on openhands index, maybe we should make a point of that.

Measured the browser schemas directly: ~2.4K tokens per call (of ~5.7K total tool schemas, o200k, current main). Against the 35.8K average prompt on durable queue that's ~7%, so schemas alone can't be a third to half the spend; the interpolation folds system prompt and history into "schemas." Measured separately, history is the biggest bucket on long runs — it grows past all schemas combined around call 8-11 — and the constant preamble is ~9K per call (3.3K system + 5.7K tools). Multiplied by the call-count difference (129 vs 69/76 on the incident repeat), that compounds into most of the gap: same work, more turns, each turn re-carrying the fixed cost and its accumulated history. Counts are per-call estimates validated against provider usage (0.97-1.00 mapped tokenizer, 0.85-0.97 fallback). Instrument is the draft PR above; per-call data available. Schema numbers measured on v1.42.1

Removing skills should help, but I don’t think it would help that much because we do progressive disclosure.

Yes, but we went from circa 60 skills to just 11 skills. So there is already a lot of context less.

Exactly, part of the task required using a browser interface.

Can you do that with the terminal? Which kind of browser interface?

So i suppose there could be a cases where the browser does make a difference, but in this case, as Opencode shows you can get away without a browser and rewards fewer tokens.

This is really on the web task that you are looking on. For example, fetching information can be done with the terminal. But you cannot understand JS parts in a browser with the terminal

• OpenHands declared 14 browser tools on all 35 calls

• It made *zero browser actions*— not a single one

• The 14 browser schemas added roughly *12,000–19,000 prompt tokens per call*(the avg prompt was 35,814 tokens with 22 tools vs. what would be ~15–18K with the 8 core tools alone)

I'm suspicious here: 12,000–19,000 prompt tokens per call . This number should be fix and constant during all the conversation (per call) as OH never used the broswer. The big problem is that this part is present from the beginning. So for every turn you pay cache on that.

Total prompt tokens for that run: 1,253,493 — so the unused browser schemas account for roughly 33*–52% of the entire token spend* on a pure backend task

Are you taking in account new tokens and cached one? The first time you pay full price then you are using cache. I agree that the web-broswer is generating a lot of costs. (not difficult to see 🙂 ) Perhaps we should think to switch to a web-broswer tool in the philosophy of Claude and Gemini CLI (and Codex)

Agents ugh: This is a backend task, so no need for browser Corrected Durable Job Queue numbers: • Browser overhead:*2,336 tokens/call*(fixed constant) • Over 35 calls:*~81,760 tokens* • Share of 1,253,493 total:*~6.5%*(not 33–52%) So partly browser, but longer system prompt and growing conversation history Still many more tokens than Pi or Opencode

Hey all. I think this is a good callout by Rajiv. But I’d encourage us not to overfit to a very small sample size. If we’re worried about this, let’s run the openhands index with pi and opencode using whatever model we like (maybe GLM?) and get a number and more trajectories to evaluate.

nod Or DeepSeek-v4-Pro for implementor agent? I’m using it a bit, it seems pretty good

I would really like that @limited-student-10747 - I just know there are lots of experiments to run across harnesses, orchestration approaches like using subagents, and different types of models. I am just trying to get my head around the tradeoffs of different harnesses for longer tasks.

Yeah, I think it’s a really good exercise and it made my think about things as well. Thanks for doing it!

I was a bit surprised and learned a lot about how we use the browser. When I rerun it with the latest agent from @modern-fireman-99755 - the results and code is much more inline with the other harnesses. (There is still an open question whether we want to be like other harnesses and should we emphasize our browser tools approach - if we have some examples or maybe I need to build them, that show off openhands for front end work, that would be useful in my demos)

I opened an issue about automating llm x harness for openhands opencode pi hermes github.com/OpenHands/software-agent-sdk/issues/4627 it requires some engineering but it could be worth while @modern-fireman-99755

Looks like 5.2 loves tool calls