https://docs.all-hands.dev/ logo
Join Slack
Powered by
On the 14th, we changed the prompt slightly for th...
# proj-agent
w
On the 14th, we changed the prompt slightly for the 
pr-review
skill. Some folks mentioned they noticed a change so I went digging through the runs of the GitHub workflow to see if I could verify that. Long story short, the available data does not suggest a change. But there's still some interesting insights: • The agent's precision (portion of suggested changes incorporated) stayed around 0.54. • Mann-Whitney flags a reduction in the average number of suggestions per PR (3.15 before to 2.2 after), but that disappears when narrowing the "before" data to two weeks before the swap. • Mann-Whitney also flags a reduction in human suggestions per PR (0.63 before to 0.1 after), and it stays significant after narrowing the "before" data. However, this has historically been trending down (-0.019 suggestions per PR per day, p=0.002) so it's probably unrelated to the swap.
c
Could you please tell what swap was that?
w
c
Ah. 🤔 With what LLM, Opus 4.6 must be?
w
I believe the action runs on Sonnet 4.5
c
What I’m thinking is, Anthropic has said for a long time that they’ve been trying to “deal with ambiguity” in prompts every LLM release since… idk, 3.7 for sure, maybe 3.5. I’m not sure what they mean exactly, I do interpret that as somehow inline with the observation that Claudes are “less prompt sensitive” than some other LLM families. On the other hand, using them a lot, I am fairly confident that Gemini 2.5 Pro and GPT 5.x are / were much more sensitive to small changes in prompts.
Inline with that idea, is also the observation that running /codereview-roasted on GPT-5.x , as I usually do, leads to fairly different results than folks running the same on Opus 4.6 (and Sonnets before)
In some sense, “dealing with ambiguity” is the opposite of faithful instruction following, I think. It kinda goes off with its interpretation of the prompt, and it’s kinda the same unless yelling at it 😅
nod 1
(on a side note, I hear that Opus 4.7 should be better at instruction following, and I see some folks disappointed that it doesn’t work for them as before)
All this is to say that I don’t know… Is GPT-5.4 affected by the absence of “Linus” in the prompt? Tests on Sonnet 4.5 are interesting, just not the same
c
Interesting 🤔 i wonder if we have the dataset so we can run things like topic modeling on the review comments from the bot
w
I think that's the next step, or at least adding more "quality" signals into Laminar. This analysis is just what signals we already have.
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by