https://docs.all-hands.dev/ logo
Join Slack
Powered by
Sadly, our COLM submission was rejected
# proj-codescout
b
Sadly, our COLM submission was rejected
😭 6
On one of the LTI channels, I found these stats for post-rebuttal scores of papers still under review: median score of 5.5 and a 75th percentile score of 6.25. So average of 6 was somewhat borderline for 30% acceptance rate 🙂 Also if the above stats are indeed true, scores shifted quite a bit post-rebuttal (see pre-rebuttal score distribution here)
a
What lti channel did you find the stats on? 👀
a
🫠
Let’s incorporate the feedback and submit it to ICML or NAACL?
b
You mean ICLR/EACL right? I don't see NAACL on ARR. ICLR decisions will come in end of January whereas it is Nov. end for EACL/AAAI. But ICLR is a better venue 🙂. The concerns are mostly just framing/positioning and not about the core claims/novelty in this submission. I am not sure about other venue choices before end of year
a
NAACL’s ARR submission deadline is October 12
b
Ok got it (for ICLR it was Sept. 24 last year for reference)
a
Sure, either one is fine
l
sorry to hear it didn’t get in this time 😞 I think ICLR is better for this one!
q
Let me know if I can help with more experiments or the paper!
👍 1
b
Ok sir thanks, we can submit this to ICLR.
Hi everyone! I have resubmitted Codescout to ICLR as discussed above.
🙌 4
Also it seems like ICLR had ~60K submissions 🙂 - for reference NeurIPS last year had <25K submissions afaik
q
Whaat wonder how much of that is slop
b
Hi everyone! I have ported over the COLM submission as-is to ICLR format in the same overleaf project as the first iteration of our draft, and created a one page summary of reviewer concerns/feedback from the COLM submission and prior ACM CAIS workshop submission here. The somewhat major/unaddressed concerns are: 1. Difficult to understand each component’s contribution to the performance gains as we change: (1) base model (qwen3 vs. qwen2.5), (2) scaffold, (3) RL recipe, when comparing to baselines. Reviewers asked to: a. Train Qwen 2.5 models with our recipe b. Train Qwen3 models with baseline scaffolds using our recipe to see if it helps → i. Without this it is unclear whether the simple scaffold is genuinely sufficient c. Evaluate Qwen3 base models and more proprietary LLMs on other scaffolds to make a direct comparison with CodeScout 2. Language-agnostic framing is unsupported with corresponding evaluations showing improvements on localization for other languages → is below result sufficient? a. We have file F1 localization scores on non-Python instances from SWE-bench Pro. Qwen3-4B-Instr. → 17.38, CodeScout-4B → 32.56, Qwen3-14B: 19.41, CodeScout-14B → 22.99. Getting function/module F1 is tricky as our ground truth parsers rely on ASTs to get edited functions/modules. 3. CodeScout only works well for OpenHands-Bash and will not generalize to other scaffolds 4. Novelty: Every component is off-the-shelf (SWE-Smith, OpenHands, GSPO, RFT). RepoNavigator already makes a very similar claim - RL with a simple tool, no closed-source distillation making the incremental contribution difficult to justify. 5. The “submission reminder” substantially changes results for frontier models, including near-collapse without it for some systems so part of the reported performance difference may be failure-to-terminate behaviour rather than localization capability itself. Also this makes results harder to interpret. a. Better if all models were compared under a standardized setup where the submission reminder was consistently enabled. One of the suggestions from COLM AC/PC was future revisions should frame the contribution as an effective open recipe for training bash-based code localization agents instead of making broad claims about specialized scaffolds not being good enough or the models being language agnostic.
I personally think concern 1 is somewhat major/difficult to argue against and will be raised if we submit the draft as-is to ICLR.
Hi everyone! I have revised the draft for CodeScout and submitted it to ICLR. I have attempted to address feedback from prior submissions described above either by (1) providing concrete evidence or additional metrics for e.g cross-language generalization, statistical significance, etc. or (2) expanded the limitations section to acknowledge interesting experiments beyond the scope of our work (for e.g. training with our RL recipe using other scaffolds/older base models). I have added 1 comment per concern in this doc summarizing what was done/not done to address each concern. If you want to review the low-level changes directly, please refer to this GitHub commit that includes the specific changes we made from COLM to ICLR excluding changes performed to port the conference format.
Our ICLR reviews are all 3 weak reject (score = 2) 😞 Not sure if doing a rebuttal will change the eventual outcome, but we can do that regardless if we want to. Or maybe we can be more optimistic and try as there is no conference after this soon enough anyways. The avg. score is ~2 it seems across most papers.
Open in Slack
PreviousNext
Powered by