This message was deleted.

Here is a link to the benchmark: https://github.com/symflower/eval-dev-quality

interesting! from the article:

• enticed the agent to “overthink” and do things it was not tasked with

• The agent immediately checked if Go and Rust were installed. With Rust being absent, it simply refused to perform the transpilation:

the new swebench instructions perform better in those fronts. I'll experiment with the system prompt to migrate the improvements there