possible replacement for the commit0 benchmark? https://programbench.com/

Thanks for sharing this sir! This is indeed a more scalable design than commit0 to evaluate the ability of models to generate libraries from scratch and will allow scaling to training data better than commit0. But I feel commit0 is more aligned with how real-world developers would want to do library generation (write the software specification + code skeleton and ask the model to fill it for you)? We won't have a binary to start with in a real-world use-case else that problem is already solved. Would love to know your thoughts on the best interface for library generation! Though one interesting use-case if this is solved would be to reverse-engineer proprietary programs like nvcc 😅

I don’t know, this one also seems to have a pretty clear spec in the form of tests right? Or maybe I misread the paper

Yes. the spec is clear in form of tests - the benchmark design is perfect for studying the ability of LLMs to generate library from scratch since it is verifiable and scalable. The only concern I have is regarding the task setup of reverse engineering a given binary instead of filling up a code skeleton. Like no real-world developer has a binary to start from when creating a library from scratch so the way in which evals are done does not align with how they are actually used downstream and this alignment is stronger for swe-bench/commit0.

What's interesting about this benchmark is that they take cheating very seriously. https://programbench.com/blog/is-programbench-impossible/

I am (openhands is) trying to get this benchmark running in our CI, wish me luck

let me know if you have issues! happy to share any pointers with our setup or review anything! the biggest delta from commit0 is less spec and more evaluating how agents probe an executable to determine behavior (exploration): in our mind an executable is technically a perfectly sufficient problem statement