Microsoft dropped Fast Context, it's very small model optimized for reading and searching code bases, by using it you can reduce token consumption by 50% on SWE tasks. If anyone has free time, it would be cool to experiment and see how well it works with OpenHands. https://github.com/microsoft/fastcontext https://arxiv.org/abs/2606.14066 This goes with to @modern-fireman-99755 post on where agents spend their time: https://pythagorai.substack.com/p/where-agents-actually-spend-their (the paper notes it works with parallel tooling calls)

Our codescout model beats them on function-level precision!

Is there an easy way to demo/use this with OpenHands, companies want token optimization, I would love to show them how they can run this as part of their working environment.

Yeah, I think we could try that

@average-processor-75195 In their RL reward function, there is a bonus if the model uses parallel tool calls. So I suppose the model can likely emit batches of tool calls. However the evaluation harness in their GitHub repository currently executes them sequentially, as the parallel execution code is commented out. But this has an influence just over runtime.

We could also do a simple thing of adding CodeScout or FastContext to our language model selector and demonstrate how it can be used to localize before switching to a bigger model for implementation

Yea, maybe we could even it work it in with the /goal work as well to switch between modesls

Thanks for sharing this! Parallel tool-calling is something we wanted to do reward modelling for but couldn't explore (it is indeed very important for complex localization I feel) Also, using only SWE-bench Verified for estimating/reporting localization performance is not a good idea in general as SWE-Bench Verified edits ~1 file on average.. In CodeScout, we report how file and function-level F1 score drops as number of Files and functions increase on SWE-Bench Pro.

Thanks for sharing this! Parallel tool-calling is something we wanted to do reward modelling for but couldn't explore (it is indeed very important for complex localization I feel)

Interesting. For CodeScout, you are just using the terminal as a tool, right? Do you think it will be trickier? Because FastContext uses just read-only tools (which are exactly the Claude ones and also used by other harnesses), which prevents conflicts, i.e., one thing less to worry about. Are you using mostly, or just, OH trajectories to train?

I think bash terminal is a standard component of every agent harness? I don't think it is trickier - our models only issue read-only commands (see Appendix F here) and OH has built in support for parallel tool execution afaik. We prompt the models to do parallel tool calling and all frontier closed-source LLMs indeed do parallel tool calling. We train with pure RL (no distillation/SFT) with OpenHands(-Bash only) as the scaffold, so our training trajs. are OH-only.

Ok interesting. Just out of curiosity: how do you fine-tune your reward function? Do you look at successful runs/trajectories and try to figure out how to capture that behavior in the reward?

Since this isn't a long horizon task, you can have an outcome reward function which is identical to your evaluation metrics (f1 scores for files, classes, functions). We tried other things like rewarding parallel-tool calling, tool formatting, turn limits etc. but they weren't really helpful/hurt performance in most cases. The ideas of auxiliary rewards we tried came from analyzing the successful and failed trajectories from RL.