Re: https://openhands-ai.slack.com/archives/C0B4Z32L064/p1782203236415129?thread_ts=1782150713.328569&channel=C0B4Z32L064&message_ts=1782203236.415129 I understand that a different LLM is automatically assigned to the code-review automation randomly (?) and we try to do A/B testing 🤔 However, in my understanding, the reviewer agent evaluation has (had?) a feedback loop, where an automatic proposal on the code-review skill prompt is opened when it decides that it needs some improvement to the prompt. ==> that seems to mean that maybe we are trying to optimize the review prompt for multiple LLMs in the same time. Is that understanding accurate? Please correct me if wrong! If that is so, that, it seems to me, is very difficult if feasible at all. Sometimes the optimizations are opposite, literally opposite because the LLM has different behavior to the same kind of nudge, even within the same model family (for example, gpt-5.0 and codex variants) What if we decide which LLMs we want for code-review, and then optimize the skill prompt separately for each of them? e.g. only 2 or 3 LLMs, not too many; each with its quirks, with different additions to the skill, depending on LLM?

Hey @calm-artist-7206, good point and I completely agree. But right now we’re not optimizing the prompt yet. @red-crayon-20970 and @wide-doctor-37830 are working on it, but we’re not at the point where it is finished yet

Hey @calm-artist-7206 -- as Graham said, the A/B testing isn't currently used to tune prompts. We're mostly just trying to understand the price/performance breakpoints of the review-bot so we can pick the best model going forward. We do hope to A/B test different review prompts, and to use the current model A/B tests as data to help us along the way, but even if the same PR has multiple models involved in the review process we've got the data necessary to separate out the contributions associated to each model.

However, in my understanding, the reviewer agent evaluation has (had?) a feedback loop, where an automatic proposal on the code-review skill prompt is opened when it decides that it needs some improvement to the prompt.

Hmm, I can't find any evidence of an automatic feedback loop in current use. Closest is the "Improve this review?" block quote, but AFAICT that just links to the per-repo code-review guides. Is there some other mechanism you were thinking of?

No… older than that. Maybe I’m dreaming, I think we had some PRs opened automatically from some evaluation of the reviewer agent runs… intended to adjust the reviewer to perform better, but now I can’t find it 😥 @chilly-france-44439 should know. Sorry, I might be hallucinating, umm, worse than my LLM!