set the channel topic: For discussion about training better agentic code search models.

Thank you, love it! Almost everything in these channels is Open Source, so I love it without "oss"

hey all, will be on in a couple minutes!

has renamed the channel from "proj-agentic-code-search" to "proj-codescout"

Hi Graham sir! Regarding this issue: I have completed Sonnet-4.5 evaluations. Unless we want to evaluate additional closed-source LLMs, we may want to close the issue. On SWE-Bench Verified, using our simple bash-only scaffold with Sonnet-4.5 is an order of magnitude better than RepoNavigator (which uses the

jump

tool) with +18 points absolute improvement in function-level F1 scores. I feel it may be worth trying out our scaffold + Claude-3.5-Sonnet on SWE-Bench Lite because it maybe comparable to LocAgent looking at the superior performance of GPT-5+our scaffold. But the Neulab litellm proxy doesn't have this model. (this is optional though)

Hi @plain-jordan-64103! I think I may have figured out a potential reason about why your training runs were dying out. The trick is to not have uv venv in the project's root directory as Ray will copy everything to initialize its workers and the .venv directory is very large O(few GBs). Also, I think I mentioned this to you before, but make sure to set

export RAY_worker_register_timeout_seconds=600

in your

run_async_training.sh

script - without this I observed that it may happen that some vLLM workers will die out without killing the trainer and essentially you will just use 1/4 GPUs to sample rollouts significantly slowing down your trainer. I debugged a similar problem in a somewhat rushed/hasty manner just before the ICML deadline and this detail slipped my mind, due to which I may have shared a stale trainer script with you (really sorry about this mistake). Here is a detailed explanation about why this fix is needed from one of SkyRL's maintainers: https://github.com/NovaSky-AI/SkyRL/issues/224#issuecomment-3246168767. After this fix, the threads/Ray workers dying out errors should become extremely rare (though not completely non-existent).

Hi all, I just pushed my fork of the benchmarks repo with support for running inference on modal for swe bench (sorry it took me so long I had a busy week). It's available here: https://github.com/taha-yassine/benchmarks/tree/swebench-modal It should be as easy as adding the

--modal

flag to the

uv run swebench-infer

command. I also have a script that deploys a vLLM instance on Modal to be used in conjunction with the above. I open a PR to add it tomorrow. Hopefully this makes the evaluation process easier @busy-branch-28834 (also if there's interest I can open a PR to upstream Modal support for inference, since it's already supported for evaluation with the same flag)

Hi Graham sir! I have a few updates and questions on the pending issues: 1. Demonstrate advantage of adding search results on SWE-Bench scores: I evaluated Qwen3-Coder-30B-A3B-Instruct on SWE-Bench Verified subset (50% instances). Adding search results doesn't improve accuracy but improves efficiency -- uses 3.66 steps less on avg. and 126K prompt tokens fewer on avg. per instance (see here). Is this result good enough/satisfactory or do we want to try another LLM (like Qwen3-32B with reasoning here)? 2. Skyline performance: I evaluated Qwen3-4B-Instruct on SWE-Bench Verified with ground truth localization results in prompt. It improves accuracy but efficiency is similar to vanilla prompting and weaker than adding search results from CodeScout (see here). Do we want to try a larger model here as well? If yes, should I proceed with Qwen3-Coder-30B-A3B-Instruct? 3. Codebase verification: I re-trained 4B instruct and 1.7B (SFT'ed ckpt) with RL and their reward curves and accuracy on test set is similar to what we had with the uglier codebase. So the codebase is logically correct after the rigorous cleanup.

Hey all, looking forward to meeting today! Let's discuss release timing

I wanted to share a quick update on the embedding search tool training runs as I will not be able to make it to the meet today: With Aditya's latest modal training script the ray-related issues were resolved helping debug the MCP-related thread exhaustion issue faster. I'm able to start training stably but the current issue is that ChromaDB has memory leaks which makes the training restart after a certain number of steps. I'm migrating to FAISS which is more stable in this regard. Based on the release date, I'll try to get the results in time

Can you also once double check if this logic is used as-is in your codebase: https://github.com/OpenHands/codescout/blob/c03d1fd74a61d8165e6a3bf35c3245f48e27f0de/src/generator/code_search_generator.py#L420-L480 (especially lines 452-456). Also just curious what is the general trend of initial reward curve (for first 50-100 steps) -- if it is a flat-line then there is likely some bug in the codebase.

Some initial results from the ablation experiments: Based on Aditya’s latest PR and a highly reproducible baseline (using GSPO) and the upgraded SkyRL-train package, I tested several algorithm variants: 1. vanilla GRPO 2. SAPO 3. Some Dr.GRPO-style length normalization methods So far, none of them outperform our current baseline on the validation set (I used a subset of SWE-bench Lite for a quick pass). Interestingly, vanilla GRPO achieves higher training rewards but does not translate to better val performance. W&B: https://wandb.ai/leon_at_work/soni_ablation_4b/workspace But one strategy that does seem promising is prefiltering a subset before training — it leads to a noticeably steeper learning curve. The experiment is still running, and I’ll share updates once we have more conclusive results.

Most interestingly, vanilla GRPO and Dr.GRPO variant is now working without errors on H200s. I wonder what changed from our earlier experiments (1.5 months back) where the training would collapse to zero rewards even with H100s (Lintang faced this problem with 14B, and this was the reason we shifted to GSPO). At this point, I have tried so many variants of the training code/algorithm, that I no longer have any idea what we were doing wrong. But the good news is that the current code is working as expected across training algorithms lolsob

Hi @limited-student-10747 sir - will you be able to make it to the meeting today? Sorry I just saw your update on the calendar invite!

Hey all! I’m going to cancel the meetings because I think we’re basically finished, but you can take the time to review the draft and comment. I’ll try to review today aiming for a release this week

Hi @busy-branch-28834 and all, I read through the paper! I left a few small comments and edits, but I think it basically looks good to me once those are reflected, we can arXiv early next week I think! Also, we need to deblind the paper, add the authors, etc

Hi everyone! Since we are going the arxiv the paper in the next 1-2 days, I am not sure about the desired affiliation you want to show for this paper for the following co-authors (it is listed as "Independent" right now) in the overleaf: @quick-honey-73477, @rich-byte-67407, @acceptable-holiday-21046. Please feel free to let me know the desired affiliations (if different) or edit the overleaf directly regarding this. Also Sriraam do you want your name to be "Bharath Sriraam Ramasubbu Raja" or some other abbreviation like "Bharath Sriraam R. R." for the preprint?

Are we working on a thread to post tomorrow @busy-branch-28834?

@acceptable-holiday-21046, @rich-byte-67407, @bulky-parrot-86531, @steep-gigabyte-86334: can you guys share your twitter profiles here? I can't find them

Hi everyone, super excited to have wrapped up and released the CodeScout paper! For more context, this work proposes an effective RL recipe for training code agents to search relevant locations in a repository using only a terminal 🚀 • Prior methods have relied on complex inference scaffolds (for e.g. using code graphs) that are Python-specific and in many cases are very expensive for deployment/RL! • CodeScout uses standard Unix command-line utilities (like ripgrep, sed) to localize relevant files, classes and functions of code, and achieves SoTA results on SWE-Bench Verified, Pro, and Lite outperforming 8-18x larger base and post-trained LLMs, closing the gap with frontier closed-source models🏆 • Interestingly, our 14B model achieves strong results using just two command-line utilities: ripgrep and sed ✨ Here are the social media posts if you are interested in reading more/sharing our work: • https://x.com/Aditya_Soni_8/status/2035008866438561874 • https://www.linkedin.com/posts/lintang-sutawika_can-we-train-code-agents-to-search-re[…]m=member_desktop&rcm=ACoAADnDWxMBQ6OdTTt3JfRYQO98LesPPFYbrXs Paper: https://arxiv.org/abs/2603.17829 Code: https://github.com/OpenHands/codescout HF Collection: https://huggingface.co/collections/OpenHands/codescout Huge thanks to all our collaborators for making this project a success 🤩

Hi everyone! Based on discussion with Prof. Graham, we have decided to withdraw the paper from ICML and resubmit it COLM. I have submitted the abstract to openreview - please let me know if you cannot find it in your openreview account (I have checked all authors' openreview IDs)! Our reasoning for this decision is that first impressions matter (our current draft is much better quality than ICML version). Also, given that our ICML review scores are somewhat borderline, we may not be able to convince everyone just through a rebuttal.

Hi everyone! We have withdrawn the paper from ICML and submitted the paper to COLM as per above discussion. Here is the PDF of the paper for reference (it includes more analyses in the appendix beyond what we have in the pre-print if you are interested in reading about error behaviours/future directions and ablation studies about alternative design of turn-limit reward).

OMG the other person presenting at amazon research day before me is the last author of the other CodeScout 😂 https://scholar.google.com/citations?hl=en&user=d-La2lQAAAAJ&view_op=list_works&sortby=pubdate

Pretty good scores for COLM! @adventurous-tiger-32098 and @busy-branch-28834 let’s start a doc and write a strong rebuttal.

Sadly, our COLM submission was rejected

Just realized that cursor no longer tries to use semantic search/vector-based retrieval via turbopuffer and simply does vanilla grep-like search - x.com/artem_zin/status/2104043933080932781 . Our position on codescout's design is not completely wrong 😅

github.com/OpenHands/codescout/issues/80: this r~esearcher has ported codescout infra to verl and is asking if we will merge PRs for it. Their numbers seem quite off as compared to the ones in our preprint, but if it is legit~ (edit: seems like their agent made some SWE errors - probably Anthropic model because it was too hard to convince that it is wrong)~, what would be the best way we could merge their contribution? Changing the code on

main

is somewhat tricky - maybe we can take their code in a branch and point from main?~