OpenAI is suggesting to use swe-bench pro instead of verified, we might want to do this for the next version of the openhands index https://x.com/OpenAIDevs/status/2026002219909427270

Thanks for sharing the blog! You maybe already aware about this but SWE-Bench Pro dataset (public version) has only 3 repositories and 266 instances for its Python subset. (the rest are in other programming languages). So the insights maybe biased since the repositories are too few if we only run evals for Python.

yeah, I think we’d run all languages if we do use it