Hi Graham sir! I have a few updates and questions on the pending issues: 1. Demonstrate advantage of adding search results on SWE-Bench scores: I evaluated Qwen3-Coder-30B-A3B-Instruct on SWE-Bench Verified subset (50% instances). Adding search results doesn't improve accuracy but improves efficiency -- uses 3.66 steps less on avg. and 126K prompt tokens fewer on avg. per instance (see here). Is this result good enough/satisfactory or do we want to try another LLM (like Qwen3-32B with reasoning here)? 2. Skyline performance: I evaluated Qwen3-4B-Instruct on SWE-Bench Verified with ground truth localization results in prompt. It improves accuracy but efficiency is similar to vanilla prompting and weaker than adding search results from CodeScout (see here). Do we want to try a larger model here as well? If yes, should I proceed with Qwen3-Coder-30B-A3B-Instruct? 3. Codebase verification: I re-trained 4B instruct and 1.7B (SFT'ed ckpt) with RL and their reward curves and accuracy on test set is similar to what we had with the uglier codebase. So the codebase is logically correct after the rigorous cleanup.

Awesome, all of these sound good! I think for 1 it’s probably good enough, and it’s the result we got so I think it makes sense to report it

Ok sir, thanks for the feedback!

OK, great work getting the experiments in!

Regarding this issue: CodeScout-4B is now available (I have verified it is the right ckpt by reproducing results on SWE-Bench Lite 😅). Would you prefer that we resolve this issue in priority as well (uploading trajectories) so that the transfer to OpenHands HF acc. is done in a single shot (it is easier to read trajs. from HF over GDrive I feel)?

yeah go for it!

Ok sir, will do that and then ping here. Just to confirm, we primarily need the evaluation trajectories right? Training trajectories are too large - but can add those as well if desired.

oh, i think training trajectories are good. wdym too large?

large in byte-size I meant. It is manageable - for CodeScout 14B, the total size of training trajs. is 7.4GB so it should work fine

ok, sg

This is achievable. I was also wondering if we really need the raw token-IDs in this data? The size will considerably go down if we remove the tokens. Also do we want the OpenAI styled messages or the raw event streams (the trajs. are stored as event streams right now but can be converted easily).

Yeah, maybe not necessary to have raw token ids

Ok sir, thanks. I will complete this task.