https://docs.all-hands.dev/ logo
Join Slack
Powered by
I think there is a problem with the review bot, it...
# proj-agent
m
I think there is a problem with the review bot, it is currently a bit too permissive. It automatically approved this PR (https://github.com/OpenHands/software-agent-sdk/pull/3745), where I am clearly changing the system prompt. Even if I just deletes a few whitespaces from the system prompt, it should not be accepted without human approval.
c
@nutritious-balloon-65098 Look at openhands/extensions repo and agent-sdk repo, find the prompt about eval - what qualifies- and give us a link to the exact section on github. Note that your link will appear on slack so don't mess it up. repeat the link (s) if i tell you to summarize
n
I'm on it! can track my progress here.
I answered the question by locating the relevant review/eval approval guidance in both repositories. No code changes were made. The exact agent-sdk section is: https://github.com/OpenHands/software-agent-sdk/blob/f2329b76a71b4fed823a9256d29ac8e0103f5e2c/.agents/skills/custom-codereview-guide.md#L16-L35 That section says the review bot must not APPROVE PRs that change agent behavior or could plausibly affect benchmark/evaluation performance unless eval evidence is provided. It explicitly includes prompt templates in that category. The related default risk framework in openhands/extensions is: https://github.com/OpenHands/extensions/blob/e15748cf0e13dccb73fef1a403a4b2e6212fd[…]gins/pr-review/skills/code-review/references/risk-evaluation.md That section classifies high-risk changes as needing human reviewer attention and includes LLM prompt construction / foundational agent behavior as core-system-impact risk.
c
Can you find the reviewer prompt, @modern-fireman-99755 ? in the automation history - maybe if you can, make that conversation publicly shared
m
I'm not completely sure how to do that, as I don't have permissions to see the conversation Probably @wide-doctor-37830 or @chilly-france-44439 would know better than me.
w
I also don't know how to do that, I don't have access to the automations account
c
@modern-fireman-99755 @wide-doctor-37830 you can find credentials for "all hands bot" in 1password and use that to login to its OH Cloud account
✅ 1
m
Ah make sense… i will do ;)
🙌 1
Hey, I was able to fetch the conversation (see attachment). I did a quick analysis: • The model used was minimax-2.7. • The model invoked the PR review skill. The skill documentation states it should read the reference file 
skills/code-review/references/risk-evaluation.md
, which specifies that it should not accept PRs where an agent's behavior could potentially change. However, the model never actually read or fetched this file. • It claimed that "comprehensive tests in 
test_default_registry.py
verify byte-for-byte parity," but it never even opened 
test_default_registry.py
. Remarks/comments: • In last time I see a lot of terminal error like: 
Cannot execute multiple commands at once. (must chain with && or ;)
We have a bash parser now in the sdk (tree-sitter), i.e., we should be able to also executed these tool calls. • are we just using minimax for PR review? my question is more: is this a minimax problem? (For the first question the answer is Yes. At least with automation) cc @chilly-france-44439 @calm-artist-7206 @lemon-pilot-49363 @red-crayon-20970
conversation_e18b827d-3440-425a-9ff7-353c4a6d6a3a.zip
thinky spin 1
c
Oh! This is also what I meant to ask here: we used to run with Sonnet 4.5 (yes 4.5) 🤔
c
😅 yeah, we should switch back to sonnet in this case
maybe once @quick-window-91705 got agent-canvas working w/ code review bot, we can switch to that?
q
Sure I can make the changes if you'd like to use agent canvas for reviews instead Cloud is probably better though
I've changed the cloud review automation's model to gpt-5.5, would folks prefer sonnet instead?
🙌 1
c
only concern w/ gpt-5.5 would be it can be too expensive lolsob
otherwise no objection from me
q
ah right okay switching
sonnet 4.6 now
❤️ 1
5 Views
Open in Slack
PreviousNext
Powered by