This message was deleted.

Some preliminary numbers for the O3-mini -- the first run on SWE-Bench Verified give us 229/500, about 45.8%. Pretty close to sonnet, and the cost per run is $295.83, which is also similar but slightly cheaper than claude sonnet ($388.03). Their official number is evaluated by running the agent 4 times and taking the pass@1 (because you can't set temperature for reasoning models to control the randomness here) -- so more eval is still running to get a final good estimate of its performance. But one thing we can be sure of for now: "It performs similarly to Claude and costs slightly less" - we will definitely make this our optional model. And it also makes "using mixture of models to work at the same time to get better performance/lower cost" more important in our roadmap

I am surprised that it actually did so well on the OH, and my guess is it just for it being absolutely brilliant on the first steps.