To improve our agent performance I suggest we impl...
# proj-agent
s
To improve our agent performance I suggest we implement this 3 P0 tickets listed in this issue https://github.com/OpenHands/software-agent-sdk/issues/2054 • The first one is about matching CC delegation APIs. It is a breaking change so I'm slightly worried? • The second one is about having a simple way of defining a sub-agent in a file. Again, matching CC format. • The third one will actually provide a big efficiency boost: declare Bash/Explore sub-agents and have the main agent use them to preserve the main conversation loop from token pollution I think this order of implementation makes sense: • with matching CC APIs, our main agent running on Anthropic models will probably have better performance • we do want a declarative way of defining the Explore/Bash tools, and matching CC shouldn't be too hard. @limited-student-10747 @modern-fireman-99755 @chilly-france-44439 FYI
c
Super-quick thoughts - I apologize in advance I'm not yet up to speed on all details of those issues you linked. (on it) There's a lot that studying CC is useful for, and it's true we definitely should be ahead, not catch up. I would like us to look carefully at the details sometimes, so that maybe we don't go too far either - I'm trying to say, poorly (sorry!) that maybe we don't always need to match CC as much as learn what we want from it, and discard the rest. I note here that the Task interface in CC comes from
Beads
(issue tracker CLI for agents) which I'm familiar with - no, I take that back, my agents are familiar with it! 😅 The essentials of
beads
seem actually very good for other models, not just Claudes. And I guess to me, your proposal means "reimplement some part of Beads in Python". Anthropic did it: they took
beads
and simplified it in some direction. (and they actually credited it this time...). Idk if we want to match Anthropic or think about what any agent needs from an issue tracker or both. I hear in your words that this is for Claudes, and of course that's true; I'm not yet worried about how it would do with other LLMs, but I want to state the reason I'm not worried: I experimented quite a bit with multi agents by using multi CLIs. In other words, delegation implemented in Python code might well be best for Claudes, I don't know; I do think that OpenAI GPTs are crazy good when they're all "main agents" (each in their CLI), and different roles/prompts, and communicate to each other. There are, it seems to me, two big directions here. Double down on delegation in Python, and respectively double down on
tmux
and
beads
and
agent mail
, which allows a single small agent codebase to replicate itself to do stuff with fresh contexts. I don't think the question is really "which to choose". Nobody knows what is best, and both are good directions! I just want to point out that this is the implement-in-Python-same-codebase-multiagent direction. Roughly speaking (again, 100-feet view, sorry I didn't yet grasp all details), I think this is one right track on which we can experiment, and it's fine for what it aims.
l
Thanks @calm-artist-7206, totally agree! I think the main thing is that (1) we want to make sure that accuracy, cost, and speed are comparable to CC. users want this obviously, (2) there is a ton of stuff available in the CC ecosystem due to the number of users, and many users like openhands for other reasons (e.g. model agnosticity, modifiability, etc.), but don't want to lose all the skills/plugins/etc. that they created. This epic is about making sure that we make these users happy.
Another consideration is that Anthropic models have been trained with the CC structure, so they'll probably perform best there.
nod 1
But I love the fact that we now have new tool presets, which allow us to experiment with different methods for implementing things, some matching CC, some not.
❤️ 1
I also am experimenting to see if the gemini file editing tools make a difference in benchmark scores for instance.
m
Just to be sure I got it correctly before I start working on these issues: I won’t break the existing delegation tool APIs. Instead, I’ll create a specific profile for Claude (similar to what was done for Gemini) within the tools module for delegation tasks. Does that sound right?
nod 1
s
@calm-artist-7206 a first quick FYI there are two "tasks" in CC:
Task Tracker
and
Task for delegation
. In those specific issues we are not interested in the task tracker (although we do have an issue open for the task tracker 😅) but in the delegation task API. Agreed that this proposal is be "a" way of implementing delegation and we should keep in mind that other ways can be better in different setups. Perhaps we can make an issue to track that? idk
l
@modern-fireman-99755 that sounds right for now, we might want to switch over to the claude API as default eventually though
👍 1
q
I'm not up-to-date on our plans with sub-agents, but I'm curious to learn. I've enjoyed using this ticket cli on my own projects as a simpler alternative to beads. It's minimal, just a big bash script that tracks a markdown file per ticket. I've added tests and ported it to other languages including python. I love our agent planner feature but I end up wanting something that I can interact with externally and isn't local to one conversation. In Ralph workflows, interacting with the tasks takes the place of my conversation directly with the agents. So what I'm highlighting here is there's some sort of task concept that exists across multiple agent runs - that tracks my overall intent for the codebase, and then agents and sub agents are doing progress tracking internally based on sub-items of that. I don't know if that's what we want to explicitly model or not, but it's how I think of my workflow now.
g
There are, it seems to me, two big directions here. Double down on delegation in Python, and respectively double down on
tmux
and
beads
and
agent mail
, which allows a single small agent codebase to replicate itself to do stuff with fresh contexts.
I don't think the question is really "which to choose". Nobody knows what is best, and both are good directions! I just want to point out that this is the implement-in-Python-same-codebase-multiagent direction.
I think we might want an abstraction on tasks and then the community can create different backend adaptors, e.g. for beads, ticket, even GUI apps like Linear, Jira, etc... that way multi-agent + humans can choose (or mix&match) their project management interface and understand each other's work + the state of the work (and likely some locking mechanism to not step on each other's toes) + build on top of another agent work Also seems like there are at least 2 scales on which this might be useful and need some different dynamics/utils, there's the intra-task scale (breaking down into subtasks and subagents that need to closely work together) and the more general project scale where work tends to be more independent / there's less coordination required at each mini-step