Today I noticed that agent-canvas was doing this text-streaming animation (btw, you probably remember that this was always portrayed in old science fiction movies... who would have guessed that computers would actually write by doing next-token prediction like they do today?). The thing is, my inference engine had already finished about 1 minute earlier, while agent-canvas was still typing. Thus, something is wrong under the hood and we are not seeing the actual tokens per second that we should.

Yeah I think unfortunately we're bottlenecked by browser rendering time. I've noticed this too so could you open an issue for it?

you seem to have some more info

I agree it’s not critical, but it’s a bit annoying. I can open an issue

low prio

@better-monitor-61654 Which model were you using? Because I have a PR for that and if I benchmark locally just the streaming part, the PR improve it. But It is difficult to test live. Or at least to see the difference live (but probably because I'm not using local models). Here the PR: github.com/OpenHands/agent-canvas/pull/1860 Here the results of the benchmark

n= 1000  main:   1.7ms   1.71µs/tok  ids=1000    PR: 0.0ms 0.03µs/tok ids=0     63x faster
  n= 2000  main:   6.1ms   3.05µs/tok  ids=2000    PR: 0.0ms 0.02µs/tok ids=0    145x faster
  n= 4000  main:  40.3ms  10.09µs/tok  ids=4000    PR: 0.1ms 0.02µs/tok ids=0    406x faster
  n= 8000  main: 171.3ms  21.41µs/tok  ids=8000    PR: 0.2ms 0.03µs/tok ids=0    725x faster

Which model were you using?

Qwen3.6-27b-mtp with llama-server

Could you try the PR if you have time. Otherwise is not a problem, I can install llama-server.

My recommendation would be to have OH test against a slow llama-server to address issues related to the agent-server blocking and its impact on the rest of the ecosystem.

Hey @modern-fireman-99755 do you need help testing 1860 or have you tested already? I think basically as long as you can run a long conversation in the agent canvas with streaming on, and you don’t notice it getting noticeably slower as you use it for a long time, this will have addressed my issue

I tested it with Haiku (which is a fast model) and creating a long conversation, and I didn't see bottleneck, i.e., it seems to work. On the other-side. I'm running with agent-canvas locally. I don't know if this is having an influence

Hey @modern-fireman-99755 I think we can reopen this on OpenHands openhands and merge

Yes new PR: github.com/OpenHands/OpenHands/pull/16164