in case anyone is still interested in swebench-pro, I audited the bench and found that a significant chunk of their problems are ambiguous or faulty: https://github.com/kimjune01/swebench-pro-audit

interesting... Are there unresolvable instances?

there certainly are, and it's ambiguous in either the spec or the codebase conventions. sometimes the spec orders the opposite of what the golden tests show passing