https://docs.all-hands.dev/ logo
Join Slack
Powered by
looking to cut a release of the software agent sdk...
# proj-agent
l
looking to cut a release of the software agent sdk, but i needed to fix a browser race condition that only affected windows tests to get the windows tests to pass, and so auto-review is not approving it: https://github.com/OpenHands/software-agent-sdk/pull/3296 would appreciate a second pair of šŸ‘€
šŸ‘€ 1
r
@limited-student-10747 I took a second look and the Windows race fix looks correct to me; is it possible the auto-approve failure is procedural because the release PR also touches eval/deprecation-gated areas?
l
Yep
So I probably need human approval but that's ok
šŸ• 1
c
I see the examples failed, though?
7 test examples ^^
My agent says there are 2 causes
@nutritious-balloon-65098 Look at this comment from the reviewer bot: https://github.com/OpenHands/software-agent-sdk/pull/3296#pullrequestreview-4324731938 But in the PR, there were comments from tests, with examples tests and integration tests failing. The bot comment says nothing about those and I think we need to fix that. Check where the pr review workflow runs from, clone yourself extensions/ repo from the same org as I think thatā€™s where it is; but also find the prompts in the custom md file in the agent-sdk repo too. Find the best place in the prompts about release PRs and add info to the reviewer to read the 3 tests workflows results, make sure they were run on the PR, look at their latest comments (each has a comment, verify what I just said) and NOT approve if they fail, ask for human review instead.
n
I'm on it! can track my progress here.
OpenHands encountered an error: **Request timeout after 30 seconds to https://fgeqkpmlsxflbwib.prod-runtime.all-hands.dev/api/conversations/1ff75961-e454-4eab-8fbf-9cf12fe535a9/ask_agent** See the conversation for more information.
OpenHands encountered an error: **Request timeout after 30 seconds to https://fgeqkpmlsxflbwib.prod-runtime.all-hands.dev/api/conversations/1ff75961-e454-4eab-8fbf-9cf12fe535a9/ask_agent** See the conversation for more information.
l
Thanks a lot @calm-artist-7206! As @modern-fireman-99755 pointed out we might want to clean up the other classes around DelegateTool as well.
(so iā€™ll fix that)
c
Oh I see we use GPT-5.5 as reviewer šŸ˜… I was skimming through its prompt and I was like ā€œcommunication to the user? thatā€¦ is a GPT piece, not a Claude pieceā€ šŸ˜‚
l
Ah. I didn't realize that this changed but it might have changed due to me changing the default model and it might have also change because xingyao is working to power the reviewer with the agent canvas and automation as opposed to GitHub actions
c
I was just coming to ask if youā€™re sure it;s a good idea at this time: because I was looking at its trajectory, and itā€™s doing strange strange things. Example: ā€¢ a lot of actions are checking git repo branches history (?!) ā€¢ e.g. for investigating the delegation issue, it looksā€¦ ugh at all repo branches about it ā€¢ not main, not release branch, unrelated branches ā€¢ e.g. it looks at this branch from April 3rd: enable delegation in agent-server - thatā€¦ probably doesnā€™t have anything to do with anything
Approx all our prompts are tested, evaluated, and dogfed for Claudesā€¦ It reacts to somethingā€¦ itā€™s a strange reaction, surely unexpected
You're also using the reviewer trajectories to test with a critic or something, right? I feel like this may confound those
Ahh but I like the end result!
It captured the validation gap without the PR I just made to tell it explicitly to look for it!
l
One of the things I'd like to do in the near future is set up multiple different models as our reviewer and see which one is better. I think this might be a good impetus to do that sooner rather than later because it's pretty easy, we just need a comma separated list of models as far as I know
ā¤ļø 1
c
I think maybe we could expand the prompt framework-y code we have, to allow for more differences in prompts in various places? šŸ¤” The tiny ā€œcommunicate with the userā€ that I noticed on the fly is among the few if not almost only difference we can currently make, in prompting, because itā€™s in a spot at the end of the system prompt - probably the only ā€œspotā€ to fill in, currently
I mean, the question is if we are willing to, because it doesnā€™t seem easy to keep track
l
Yeah, Iā€™d love to A/B test prompts as well!
I think itā€™s not hard to keep track with logging
šŸ„³ 1
c
Hey @limited-student-10747, wondering if this release is ready yet? @millions-afternoon-47582 was hoping to get https://github.com/OpenHands/software-agent-sdk/pull/3322#top in the next release to unblock agent-canvas, so if this release not gonna ship very soon, i wonder if we can just merge main and re-run all these tests
thankyou 1
c
Examples were failing again ^
Maybe we could update it from main and rebuild?
šŸ™Œ 2
l
itā€™d be great if you could help get it in, iā€™ve been a bit busy today
6 Views
Open in Slack
PreviousNext
Powered by