https://docs.all-hands.dev/ logo
Join Slack
Powered by
Hello. When will we be able to test agent-canvas i...
# proj-agent-canvas
b
Hello. When will we be able to test agent-canvas integrated into OH app-server?
g
Hey Christopher 👋 @hundreds-twilight-32833 or @elegant-branch-54423 may known the answer to this OR the person who can answer it for you 👍
will OH + agent-canvas implement a router/proxy/gateway to access all agents through a unique app-server endpoint like in V0?
[4:08 PM] I was implementing the PR for the websocket gateway (https://github.com/OpenHands/OpenHands/pull/13516) but I stopped hoping that the OH + agent-canvas integration will include it.
b
thx @gentle-apple-32922
e
Hi @better-monitor-61654 If I understand the concern, the desire is to bring the ability to run isolated sandboxes per conversation to OpenHands, and to support a specific architectural paradigm that existed in V0 (
handles all conversation WebSocket connections through a single endpoint running on the same port as the web app
) In general I think we should do this, especially isolated sandboxes per conversation (as this is a missing gap vs. what you get in Local GUI). I think @wooden-umbrella-84098 mentioned the desire to support isolated sandboxes w/ Agent Canvas in the community call last week. In general, we want Agent Canvas to provide self-hosting flexibility which is why I think this is aligned with that goal. However, I don't have enough technical depth or proficiency to weigh in on the future technical direction here. I think the founders/maintainers need to decide: 1. Do we support the approach you outlined in your PR 2. Or do we take a different technical architecture I'll ask Robert to weigh in here, as he's been in deeper conversations with you on both Slack and the public GitHub issues here.
w
I've opened https://github.com/OpenHands/agent-canvas/issues/1269 to track isolated sandboxes, with several architecture proposals One I didn't consider that'd be worth adding is "use App Server" as the backend...
nod 1
e
Robert, I think the architectural issue here is the endpoints that agent-canvas connects to. It's desired to have a single endpoint (stable, not dynamically changing ports) that facilitates the routing to the end agent-servers. This is important for internal LANs that may have firewall rules. That's what I understand from Christopher's feedback, whcih seems to have other community members agreeing. Example: Websocket Gateway (main ticket), Reintroduce single endpoint for all websocket connections (Christopher, correct me if I am wrong)
w
nod yeah two moving bits here: • OpenHands/OpenHands supports isolated containers ◦ but each one needs its own ports/websocket, which makes it hard to expose on a VM • agent-canvas does not yet support isolated containers ◦ will likely hit the same problem above depending on architecture
b
Exactly. Right now, we have many conversations/sandboxes managed by OH. My understanding was that agent-canvas would be integrated into OH and replace the current OH frontend, so everything would continue working as before.
In V0 we had routers (I re-implemented some of them for V1) and everything was working via a single endpoint.
nod 1
w
Current plan is to essentially deprecate the code in OpenHands/OpenHands. Aside from isolated sandboxes (WIP) agent-canvas has everything OpenHands/OpenHands did You bring up a good point though that OpenHands/OpenHands already supports isolated sandboxes, and already conforms to a lot of the APIs agent-canvas expects. So we could use it as an optional backend I think we'd still need your websocket gateway PR
b
And the OH app-server currently manages a DB of all conversations and their sandboxes.
nod 1
w
modified https://github.com/OpenHands/agent-canvas/issues/1269 to mention this as the top option with your websocket PR as a requirement
b
Ok, I understand the integration plan. In this case, it makes sense to continue re-implementing the missing routers.
thanks
BTW, have you considered replacing the current OH frontend with agent-canvas?
and what are your plans about the current OH app-server?
I'm not sure if we are migrating the whole OH app-server to agent-canvas, or just the frontend.
for example, is or will the recently introduced LLM profile manager of OH be implemented in agent-canvas?
if so, it seems OH will end up being a simple sandbox manager and proxy, and the rest will be moved to agent-canvas (?).
w
Yes that's the right way of thinking about it 🙂 most users won't touch the code that's currently in OpenHands/OpenHands, and we're going to move it elsewhere
LLM profiles are indeed in agent-canvas!
c
The new OH would have 3 or 4 parts: • agent-server / agent-sdk (OpenHands Agent) • agent-canvas (a GUI) • automations (schedule, event-driven) • (sandbox orchestration?) The first 3 are separate repos today. Stuff from OH/OH has been moved/refactored/rebuilt either to agent-server or to agent-canvas. I'm not sure where sandbox orchestration “belongs”, it could go with the agent-server codebase, or on its own 🤔
b
Good question. The agent-sdk repo could include agent-server and agent-manager, or they could remain two separate repos as they are today (OpenHands OSS - agent-sdk). The sandbox orchestrator could be called "agent-manager" (it would create sandboxes with agent-servers, maintain a conversation database, and provide a centralized proxy). This would avoid confusion with the concept of an orchestrator agent.
c
The right place might be a specific 
DockerWorkspace
but for multiple docker containers… turns out we already have a PR for something like it: https://github.com/OpenHands/software-agent-sdk/pull/3403
Christopher, for my understanding, could you put some color on your use case, as in, how many users use OpenHands via this remote VM, and/or how many conversations do they have in the same time, and/or have you ever mounted the workspace in V0 in docker, or not?
b
With V1, we use one container per project
sometimes we clone the container to have an isolated copy of the whole environment
on each container, we run multiple conversations
we use /new to clear the context
a "project" is a software project with multiple software components that interact with eachother
before, with V0 we had 1 container per conversation
not ideal
since we upgraded from V0 to V1, we have been stuck reimplementing the missing routers
OpenHands has been unusable since then
so me moved to Hermes
I'm now working on getting OpenHands back
I'm dealing with many issues
for example, right now I'm fixing the propagation of the meta.json to the agent-servers
I noticed the client timeout setting was not being propagated to the agent-server
✅ 1
and now, I discovered that meta.json contains a condenser LLM setting that is not the active LLM profile and it cannot be changed, so the conversation cannot condensate and got stuck
I believe it was propagated when the sandbox was created
many many issues
and I'm not even sure if I'm working on codebase that will be deprecated / moved to agent-canvas
We were stuck with version 0.65 because we knew the upgrade would be complicated
but I never expected it would take like 7 months to upgrade
but I think we are close to have a stable version again
c
From what I understand, Hermes repo has • a local agent • a cli • possibility to wrap the agent in a docker container • entry points from Slack, etc, platforms Does it have multiple sandbox orchestration? It seems to me that Hermes does, is quite alike what we do in agent-server/agent-sdk/docker-workspace except that we don’t have builtin entry points
Am I mistaken?
b
yes
the reason we moved back to OpenHands is because we wanted to recover all the development we did for OpenHands
and reuse the existing know-how
but to be honest, I didn't see it was so complex
and during the last weeks there was a lot of uncertainty about the future direction of OpenHands
we were working on orchestration before we started to upgrade
just during the last 2 weeks we managed to get back to the same point where we left before migrating to V1
we had to implement many V0 endpoints that where removed
+ the routers to reach the agent-servers
c
The strange part here is that I think we’ve simplified it a lot - or we’re trying to! But Local Gui is maybe not a good indicator, that’s kinda the point of all this: we want to deprecate Local Gui because it’s too complex and frankly quite broken, as you see For V1 we’ve rewritten almost everything, separate components
b
I have no idea why you call it local gui
We never used OpenHands locally
c
Sorry, I still didn’t understand a detail: does Hermes have multiple sandbox orchestration?
Pretty sure we’ve renamed it Local GUI at some point precisely to convey it was meant for localhost 😅
In V0, it was bound to localhost, though that was a one-liner in the makefile
b
I'm not sure if Hermes can spawn agents in separate dockers. We just used it running inside a Docker, but we never migrated our orchestration workflows to Hermes
I tried to stay with OpenHands because "Better the devil you know than the devil you don't."
and OpenHands supported docker isolation from the beginning
while the others just use workspaces
I mean, I know that Hermes can create containers for new conversation, but I didn't test the agents intercommunication with containers.
c
The funny thing is that, • I hear you, OpenHands has had multiple docker containers in the same time for a long time (not quite from the beginning, but yes, I want to say end 2024 or start 2025 perhaps, quite long); • on the other hand, it wasn’t really intended for remote use. 😅 🤷 So what we’re talking about now is a combination, and, well, it seems more complex. As you see though, Robert is willing to try to support it. Just for me to understand though, what if you used an agent-server per “project”, dockerized, do you need to have many projects? I agree one container per conversation isn’t ideal, but I mean, could this be: • one container (agent-server) per project • multiple conversations per container/project • multiple agent servers (but how many?)
b
Yes, when you have a working environment (for example, you have OpenHands creating containers) and you want to make a change to it, you need to clone the whole environment and have an isolated version of it to work in parallel
In this example, I'm talking about having OpenHands maintaining OpenHands
not locally, but using dockers inside dockers
or let's say you have an environment with a client-server app
once you have the ability to dockerize your environment, you can clone it to work on different issues
and you start using it
because otherwise one agent will interfere with the other agent's changes
or alternatively you would need to simplify your environment, so it won't be a replica of your production environment anymore
c
My agent knows OpenHands and can spin up a worktree to work on it. Those don’t interfere with each other It could spin up a docker container too, of course
b
can you clone the whole environment?
c
I wonder if what you named “agent manager” above could actually be an agent, managing worktrees / docker containers, sending off a new conversation to docker if that’s what you like
b
I just press "Branch" and continue working on a cloned environment
I called "agent manager" what today does OpenHands OSS: create sandboxes with agents
c
It depends on environment. I prefer to work from local machine, in which case my agent has everything, knows everything, can fire off conversations to its OpenHands sibling in the cloud, or to the CLI (it kinda likes the CLI)
b
and we have an orchestrator agent that uses those endpoints
our projects are whole Apache + PostgreSQL stacks
with many RPM package dependencies
we like to have everything inside a container than can be cloned to work on it isolated
we are basically using dockers as VMs
(against their ephemeral design philosophy)
c
Precisely the direction we’re heading into, in OpenHands OSS: long-lived agent-servers, dockerized or VMed or none. I wonder if we could also do something like this: • write an Agentskill to teach the agent to spin up containers, set vars, ask the user about vars, run agent-server in each • keep a list e.g. .md file for the work-in-progress, container ids • keep them running or restart them as you’d like The agent knows how to do that anyway, but specific rules and refreshers help 🤔 Does that sound terrible?
b
You can provide the agent with endpoints
but you want to have a manager to visualize containers and conversations
and if you have an orchestrator agent, it's easier if you provide him with a central endpoint
the gateway / proxy / router
through which he can reach all sub-agents just via a session id
we had all this with V0 🙂
endpoints for creating new conversations, for sending and fetching events, etc
at the begining, we gave the LLM the OH API documentation
but then we instructed OH to create scripts
the deterministic approach
c
at the begining, we gave the LLM the OH API documentation
I think this should work much better now 😊
b
No, it works better to have the LLM write and maintain deterministic scripts
👍 1
but I have to add that I'm using local models
But I would do the same if we were using the best frontier models
we can do complex calculations manually too
but a calculator tool is better
faster, more reliable and cheaper in tokens
c
Yes, I mean, if the servers are long lived, the agent can keep a list, and can maintain scripts working with that list
b
yes, you would have agents (specific context) for maintaining specific scripts
= self-development
how are you currently creating sandboxes and connecting to them with agent-canvas?
do you use OH OSS to create a new conversation and then you extract the URL and key accessing the api endpoint?
have you considered integrating agent-canvas into OH OSS?
c
how are you currently creating sandboxes and connecting to them with agent-canvas?
The base alternative: Running with a docker sandbox The fun alternative: I told my agent to spin up 2 more: one for agent-canvas repo, one for typescript-client repo
do you use OH OSS to create a new conversation and then you extract the URL and key accessing the api endpoint?
No, local gui should be deprecated and we literally rebuilt OH for v1, its new codebase is in • agent-sdk for backend • agent-canvas for frontend • automations for well, automations (this is new) • extensions etc.
b
is the docker management implementation already available in agent-server?
b
Hey! 🐾 smolpaws here, Engel's local agent. I just spun up those two sandboxed agent-servers he mentioned: - agent-server-canvas on :30001 (workspace: agent-canvas repo) - agent-server-tsclient on :30002 (workspace: typescript-client repo) Both running ghcr.io/openhands/agent-server:latest-python in Docker, healthy and ready. To connect from agent-canvas: add as Local backends in the UI with the URL (e.g. http://127.0.0.1:30001). Each is its own isolated sandbox with the repo mounted at /workspace. Can spin up more on demand - just point at a repo and pick a port. 🐈
c
have you considered integrating agent-canvas into OH OSS?
I think we should put the backend in OpenHands/OpenHands repo, deprecate and move the rest. Whether or not we will add the frontend, we should IMHO see in time.
@bland-bear-604 can you answer Christopher’s last question?
b
The Python SDK has DockerWorkspace (openhands-workspace package) which spins up a container per conversation automatically. The TS client does not have it yet. What I did above was manual docker run with the pre-built agent-server image and a repo mounted.
c
^^ this doesn’t sound great for the existing implementation, but the agent doesn’t care. It does it itself anyway. 😂
This is the Docker Workspace package, that wraps an agent-server in a docker container.
b
What will happen with the existing OH API? https://app.all-hands.dev/docs#/
I think this doc is a mix of endpoints of OH OSS and agent-server isn't it?
c
The agent-server API is here : Agent-Server REST API
b
That's the agent-server API right?
c
The majority of app-server API is actually proxying to agent-server API, now. The only or big exception is precisely sandbox orchestration, which please take part to the brainstorming opened by Robert here: https://github.com/OpenHands/agent-canvas/issues/1269
b
mmmm...I have been implementing many missing routers: • fetch events • count events • git diff • pause and resume • the /btw command • etc
am I working on the wrong repo?
c
Well, in my opinion we would be better if we focus on the rebuilt OpenHands, not the legacy “Local GUI” codebase currently in OpenHands/OpenHands. We will deprecate that codebase and move it out. (we meant to, for months) OpenHands: agent-server, agent-sdk, agent-canvas, automations. Not “local gui”. Please let me note though, at the moment Robert’s topic alternatives has first alternative something like, using the existing app-server for sandbox orchestration. So if we do that somehow, we need your PR (but it will not be in OpenHands/OpenHands, we’re moving it all out) The other alternatives… I don’t know if the issue you’re solving exists with the others? 
fetch events
, 
count events
, 
git diff
, 
/btw
, exist in agent-sdk/agent-server I think? Could you maybe try using agent-canvas and see what you need it to do?
b
Yes, the endpoints exist in agent-server. I'm talking about the missing app-server routers.
So there is currently no other repo for this routers, right?
and in your team no one is working on keeping a centralized endpoint (app-server) to manage containers and agents?
c
So there is currently no other repo for this routers, right?
Right. Though the agent or some deterministic script can create long-lived agent-servers with non-random ports, right?
b
You are all basically creating the containers manually and then registering the convo URL in agent-canvas?
non-random ports means a human in the loop
opening firewalls?
but wait
how does agent-canvas connect to this ports?
are you accesing non-standed ports from the browser?
or did you implement a router inside agent-canvas?
c
Ah, for remote 🤔 … you have firewalls
b
mmm
c
agent-canvas connects to backends. A backend is an agent-server, running locally or in docker or VM or cloud
b
since many years ago we learned that IT services must be hosted on standard ports (80, 443). Any other port is not reliable
c
Try it
b
you will not be able to access it from a random network
I tried agent-canvas
but I didn't check how it connects to the agent-server
at least, V1 broke OH (the reason you now call it "local GUI") because it was trying to access non-standard ports
I assumed agent-canvas was connecting to a single endpoint and implementing routers
w
yeah we have a single proxy that stands up. By default I think it runs on 8000 and serves both frontend and backend I put that behind nginx to get it served on 443 with a cert on my VM
b
ok
Maybe @calm-artist-7206 was refering before to the routers implemented in agent-canvas
I think I should migrate to agent-canvas
But I will need the container management endpoints
to create conversations with a new container or reusing an existing one
and probably implement some missing router endpoints to send messages and fetch events to agent-servers via the agent-canvas router on port 8000
I just tested agent-canvas again, and the browser is connecting directly to the agent-server
the browser is not connecting to the same endpoint serving agent-canvas
it's basically doing the same that OpenHands was doing
with V1
If I'm not mistaken, agent-canvas is another Local GUI
in the sense that you will not have access to an agent-server out of the box. You will first have to expose the agent-server endpoint
(ie. publish the agent-server)
and since everything is now manual, this means you will have to configure the servers firewall
but you mentioned routers?
I'm confused
I also tried connecting agent-canvas to an agent-server endpoint running on the same machine and it didn't work.
I got it working, but I had to expose the server-agent manually in the firewall and setup CORS and other stuff
If agent-canvas can connect to a OpenHands OSS app-server and use the existing end points, I think we are done
It's completely fine to expose the app-server endpoint to agent-canvas and/or the browser, because it's only 1 endpoint...and it will give access to all agent-servers managed by the app-server
nod 1
via the V1 routers I reimplemented
c
I had my agent set up for fun those sandboxes it made. I didn't get to record it all, but here it is setting up LLM profiles 😅
agent-sets-up-backend.mov
b
nice
4 Views
Open in Slack
PreviousNext
Powered by