Hey all! I think with @calm-artist-7206’s code review automation and the VM one that @modern-fireman-99755 set up we’re starting to get a fair amount of noise in repetitive code reviews. I think we can maybe try to consolidate? I’m open to the best idea on how to do this, but my suggestion would be that we: 1. make sure that the automation that Vasco has created is pushed somewhere on github, probably here: github.com/OpenHands/extensions/…/github-pr-reviewer 2. we refine that and make it better together and make sure it’s constantly deployed on the oss-openhands VM 3. it’s still OK to use AI to help you review of course, but for other AI reviews we make sure that a human has generally checked that they’re not duplicated with other reviews before posting.

😅 Ouch, yes. That PR has 2 of mine and 3 of Vasco’s

I think yours was definitely good when the other reviewer was down! Also, it seems that automations still fail sometimes so I’m trying to debug why reviews are failing, but after we fix that hopefully it’ll be robust. If you have suggestions for how to use models I’m also very open (as long as cost is reasonable 🙂)

Yes, may I ask what LLMs are we currently using, the AH team working on the repo?

A couple of point: • at the moment we cannot switch between models and or profiles in automations. But theretically we could assign a specific llm for a specific Pr (for example big one to some model and small one to another) • Now the review bot is posting a new msg every review (and this seems to make noise). Perhaps we can fine tune it to rewrite the message write the new review (but always putting a summary of previous review so we don’t lose context) • We were using gpt-5.6-sol via subscription but i changed to glm-5.2 the other day. This is because we reached the max usage with gpt

In other words, it might have been useful to the project, that I was using a Gemini at some point; or a GPT-5.x for a long while, when most of the implementor agents were using Claudes

at the moment we cannot switch between models and or profiles in automations. But theretically we could assign a specific llm for a specific Pr (for example big one to some model and small one to another

It seems like we should make this more flexible, we have an issue open for this i believe, I can take a note that we should add this.

Now the review bot is posting a new msg every review (and this seems to make noise). Perhaps we can fine tune it to rewrite the message write the new review (but always putting a summary of previous review so we don’t lose context)

I think it’d be better to post a new message.

I think the automation has a profile field right now

Yeah, but you can only use one profile, not switch between multiple ones. But I guess with @millions-crayon-32880’s model routing it should be easier.

In fact, just to note, on the side point I was making above, I sometimes think we can benefit from having the prompt(s) linked somewhere, e.g. a link from the PR description. Sometimes I’m curious why an OpenHands agent made something weird, and LLM and prompt can definitely help get some intuition of it Or, just to get a feel of how likely a big blunder is

Yeah, it’s an interesting idea but I’m not sure how we’d implement it. Open to suggestions!

If it’s markdown… and it probably is easily markdown, maybe we could ask in (PR template, agent PR tool) to add the prompt in a file e.g.

prompt.md

in the temporary

.pr/

directory 🤔

Interesting! I think “ask the agent” adds extra agent steps and may not be super-reliable, so if there’s an alternative that’d be nice, but I’ll think about it more

One note about the noise in this PR reviews: I seem to see that the last 2 were posted from

all-hands-bot

one after the other, without any PR author changes in between. That doesn’t seem useful? I suggest we could do something like this rule my automation had:

NO UNADDRESSED ENYST REVIEW: look at @enyst's submitted PR reviews and find the most recent one.
   - If enyst's most recent review was made against the CURRENT head SHA (review.commit_id == head SHA), enyst has already reviewed the current code and it is waiting on the author — this is an UNADDRESSED review → SKIP.
   - If enyst has NEVER reviewed, OR the head SHA is newer than enyst's most recent review's commit_id (the author pushed new commits since) → PROCEED to review.

In other words, skip if no commits happened on the PR since last review

Yeah, we should fix that. BTW, just as a note surprisingly this is why the reviews got stuck: The agent got stuck at 06:25 because the LLM (

openhands/glm-5.2

) fell into a repetitive tool-error loop it couldn’t escape. The last ~7 observations before the

stuck

transition were all the same kind of failures, back to back: 1. `Cannot execute multiple commands at once. Please run each command separately…`(terminal rejected a multi-command call) 2. `Invalid path parameter: /tmp/review_body.md. File already exists…`(file_editor`create`on an existing file) 3. `Parameter 'file_text' is required for command: create.`—*repeated 4 times in a row* The model kept calling

file_editor create

without the required

file_text

argument, ignoring the error each time. The stuck-detector counted these repeated identical failures and flipped the conversation to

stuck

. The poller then saw

stuck

and posted the warning comment on the PR.

It might be useful, if @modern-fireman-99755 can, to enable logging of LLM requests/responses, just to make entirely sure that we do send to GLM the error… and maybe to see any reasons the LLM gives for repeating…