https://docs.all-hands.dev/ logo
Join Slack
Powered by
Re: <https://github.com/OpenHands/software-agent-s...
# proj-agent
c
Re: https://github.com/OpenHands/software-agent-sdk/pull/4299 Are we sure about removing QA changes, I thought this was actually testing live the changes of the PR? (I don’t mean its eval, the tester workflow itself)
@modern-fireman-99755 this was not a reviewer, it was a tester agent
does your agent have and use the prompt for live testing?
m
Ah sorry
No I checked it was a reviewver
c
I want to say the tester was good to have. It really verified like, read PR description, reproduce on main, apply changes, run the exact PR and flows, see the results
I don’t mean the reviewer
This… just a sec
Before Xingyao made this, we were manually prompting the agents to “read carefully etc, run the PR yourself, enable logging/etc to see results, post the logs and a summary etc in .pr/ , note that you find this there, that here, etc”
My perception is that we honed that prompt on the fly, then Xingyao made the skill and we continued to improve its behavior; then Calvin and Xingyao did some evals etc, and we ended up with this plugin. I think it was good, and replacing it leaves us … idk, back to manually prompting for live testing?
I mean, we can use the plugin, but idk. It was nice that it ran in a sandbox, too, and that is was unintrusive
Sorry, but it is nice. It’s presented quickly here: https://x.com/xingyaow_/status/2069448427923021989?s=61 I think maybe: • the reviewer should know about it • pay attention to comments left by this verifier agent • I don’t think they need to be in the same environment or run; could be left separate as they currently were • if we can still use the Cloud for this, we… could keep it and run it there?
l
IMO we don't need two separate reports if we are running both every time, it seems like too much noise? We could just do QA and review in the same automation
c
Yes you’re right, it shouldn’t be both in the same time. The review changes and should change quite a lot and maybe we can have more than one, while this QA agent run doesn’t change as much 🤔 It tests the PR itself, per its own terms, so it’s not needed • every time a PR changes because of breaking other things • every time a review adds/changes because we try e.g. mean Linus Actually… what if we make it part of Evidence section? So instead of posting yet another comment, it edits PR description and does its thing
To put it differently: they shouldn’t run in the same time, we can separate them
l
Interesting points. One other feature I’ve considered is allow automations to run again in the same conversation. If we do that we could do: 1. QA agent 2. review agent initial run 3. review agent after first revision 4. review agent after second revision
So it keeps all of the context the following reviews, I think that could help a lot with the agent continually requesting new things and also to save cost
m
One other feature I’ve considered is allow automations to run again in the same conversation.
We can already do that. e.g., the slack bot run always in the same conversation if you ping him in the same thread
l
got it so maybe we can have the review bot do that too?
nod 1
m
theoretically we could do that, yes. The automation will become a little complicated.
c
FWIW I tried that, on GitHub bridge for my agent, and I loved it. I think it was smarter and likely used less tokens overall
m
on GitHub bridge for my agent,
what is that?
c
Like it identifies the Slack thread id, it identifies the Github repo-id, where id is issue or PR, I think?
One sec
@bland-bear-604 make a page on enyst io as you know, to document the behavior of GitHub ‘app’ or bridge that you have
🐾 1
b
🐾 Done — page is live: enyst.github.io/smolpaws-github.html (also linked as a card on the home page, next to the Slack/Discord/WhatsApp ones). It documents exactly the behavior you were describing to Vasco. The GitHub ingress isn't a socket bridge like the others — it's a Cloudflare Worker (signed webhooks + a 1-min 
@smolpaws
notification poll). The bit you loved is the conversation mapping: 
buildConversationId()
derives a deterministic id from the thread — 
github-<repo>-<issue/PR number>
— so every event on the same issue/PR resumes the same conversation and keeps its context (the PR it already read, the diff it already ran), instead of starting cold. Page also covers the trigger rules (mention-or-own-thread, self-loop guard, fail-closed allowlist), dedup, the queue→reply flow, security boundaries, and a table contrasting it with the socket bridges. All source-grounded from 
apps/github/
. 🐾
c
Right, so it does this
The SDK allows to send conversation id, instead of receiving it
2 Views
Open in Slack
PreviousNext
Powered by