Pretty good scores for COLM! <@U084QDLCBS4> and <@...
# proj-codescout
l
Pretty good scores for COLM! @adventurous-tiger-32098 and @busy-branch-28834 let’s start a doc and write a strong rebuttal.
👍 3
🙌 4
b
I have started a draft here: https://docs.google.com/document/d/1E12PZ0nanmfIHbLhsFz4EZ0Fjm0nlII4LTIq0b60-AY/edit?usp=sharing. The responses still need to be filled. Will send a message here once it is ready for review.
I have finalized most responses from my side. There are a couple of things like statistical significance/more metrics that I am working upon. Please let me know what changes are required. One thing that many reviewers ask for is showing zero-shot language-generalization of CodeScout. This is beyond the scope since we trained simply using Python, but I will try and see if it happens on SWE-Bench Pro after all other work is done. A practical problem here is ground truth patch processing - figuring out edited functions and modules is hard since that still requires AST parsers.
All responses are done - regarding cross-language transfer, my initial results on 180 randomly chosen instances show that codescout-4b has 29.95% file F1 on non-python instances from SWE-Bench Pro (languages covered: go, js, ts). For 186 randomly chosen non-python instances, Qwen3-14B has around 21.22% file F1 and CodeScout-14B has around 25.73% file F1. Computing function and module F1 is hard since our ground truth processing scripts are python specific. I am not including these results in the rebuttal for now because we have around 450+ instances in total so I don't want to overclaim, but most likely our models do generalize across languages. The evals are taking really long to complete due to network issues in babel I guess. git clone is taking a much longer time
I have submitted the responses in the interest of time (the deadline is 1 Jun AoE). Apologies for being behind on schedule for the rebuttal. The only nice-to-have experimental result that we did not include is cross-language generalization and we can report findings from the above setup a bit later if we want to.
Regarding further questions from reviewers, I have added responses here. Also the cross-lanuage generalization results on non-Python set of instances from SWE-bench Pro are ready - it generalizes to Go, JavaScript and TypeScript for files.