This message was deleted.

I'd love to try out your evaluation tool as well. I've been using SWE-bench out of the box (using the scripts from SWE-agent repo). But can't get it to work running in Docker. If you're interested I collect some data on the benchmarks I've done with comparissons to the results from the auto-code-rover benchmark results in this sheet. Mostly been focusing on context retrieval atm.