I have a proposal, right now we have two methods to connect to the backend: 1. “cloud”: openhands cloud 2. “local”: connect to an agent server There are two problems with the “local” setup: 1. despite the name “local”, it should actually be able to connect to a remote agent server running on another machine 2. currently this is very messy, and actually calling an agent server on another machine doesn’t work, or doesn’t work consistently because various places in the codebase do things like assume that the backend is same origin, route through the canvas and use canvas-specific endpoints, etc. My proposal is, we create three connection types: 1. “canvas”: this connects to an agent server that is brought up together with the canvas 2. “agent-server”: this connects to a stand-alone agent server brought up separately from the canvas 3. “cloud”: connect to the cloud I already have a draft of this here, but I might want to redo it to make it cleaner: https://github.com/OpenHands/agent-canvas/pull/951 @fierce-account-55183 and @wooden-umbrella-84098 if you have any thoughts I’d like to hear them, I’d probably want to merge this into both main and the release branch, because certain commands like

npm run dev:frontend

are completely broken now.

This makes a lot of sense to me

I wonder if these three could be basically: 1. local (in-house aka handled by canvas) 2. remote (an agent-server somewhere) 3. cloud (an agent-server on the cloud)

This feels like a nice usability improvement

I’d be good with “local”, “remote”, and “cloud” (it’d also reduce renaming in the code base a bit).

@calm-artist-7206 resolved my critique, these are known terms and are meaningful to users, deviating adds overhead costs to the UX and their education.

@nutritious-balloon-65098 open a PR making the minimal set of changes to make this possible, and QA to make sure that both the local and remote paths work as expected (local running from

npm run dev

and remote running by spinning up a separate agent server). Add animated gifs to the PR description by uploading them to the “.pr” directory demonstrating that you can go through the entire onboarding process in both cases, spin up a conversation, and get a response from the agent.

I'm on it! Graham Neubig can track my progress here.

this is semantics, but ideally there is very little difference between "local" and "remote"--local is a special case of remote But if it's causing confusion, OK by me

The biggest difference is whether the backend URL is the same as the canvas URL. We have a bunch of things that depend on this (such as whether to rewrite the backend URL, or whether to expose APIs that are only exposed in the canvas, not the backend)

ahhh yeah. I wonder if we should just force "local" to use a different port instead of trying to glue it together

It’s an option, but then it wouldn’t be able to call the canvas_ui tool, and also would be subject to CORS issues.

we should be able to just disable CORS on the backend--everything is API-key authenticated anyways. not sure what we're doing with the canvas_ui tool but we should just be seeing that come over a websocket and responding to it IMO any remote backend should behave just like the normal "local" one

SGTM! One other thing is URL re-writing. Currently we re-write the URL to match the window URL of the canvas for localhost-like URLs. Should we just remove that?

which URL are we rewriting?

AFAIK we’re writing the backend URL (like “localhost:18000”) to the URL of the canvas (like “localhost:8000" or “my-virtual-machine.url.com”) and then proxying through the canvas. That means that if you run “npm run dev” on a remote machine then you’ll still be able to access the backend, even if you enter the URL of the remote server.

Covered live, but to close the loop, here's what I'd like to see: I can run

agent-canvas

and be automatically connected to the backend I can run

agent-canvas

or

docker run <http://ghcr.io/openhands/agent-canvas|ghcr.io/openhands/agent-canvas>

anywhere (as a side-process on my computer or on a VM), and connect that manually as a backend Architecturally, my suggestion is: • on frontend start, if there are 0 backends, auto-create the local backend using VITE env vars • from then on, there is zero consideration of "local" vs "remote" I would also suggest we call the two categories "agent server" and "agent cloud"

Thanks @wooden-umbrella-84098! A few clarifications:

on frontend start, if there are 0 backends, auto-create the local backend using VITE env vars

In the meeting we said that the behavior would be “populated from the agent-server window URL, with the default port”. I think that might be better because you don’t need to manually enter the URL at startup time. But I could be convinced otherwise.

I would also suggest we call the two categories “agent server” and “agent cloud”

For my first PR I’ll stick with “local” and “remote” and then send another PR to rename. It might mess up the GUI a bit to have the name include “Agent”, but I could take a stab.

agreed on both!

@wooden-umbrella-84098: do we want to update API keys when VITE_SESSION_API_KEY changes, and if so for what backends? if we had the local/remote distinction then we could do it only for “local” backends, but if we remove the distinction the logic is not clear, maybe just never update?

Yes

Further, we can assume calls to app.all-hands.dev will not be blocked by CORS if they use an API key, right?

I think this was recently fixed--for now we have a CORS proxy

OK, sounds good. “for now we have a CORS proxy” is problematic because it means you need to have a local backend running to connect to cloud, which complicates things significantly. i just about finished checking my pr with local backends, and will test on cloud next to make sure it works without a local backend selected