@nutritious-balloon-65098 check this issue: https://github.com/OpenHands/agentic-code-search-oss/issues/46#issuecomment-3696951032 I’d like to try to add module-loc and function/entity loc results for the baseline “locagent” and “swe-fixer” methods. Search online for the papers (use the “src” URL of arXiv to download the source) and clone the repos from github. See if their localization results are available anywhere. If they are, run our evaluation code over their localization results. If they are not available then figure out if they even make these predictions.

Summary I investigated the baseline methods (LocAgent and SWE-Fixer) for module-level and function/entity-level localization as requested in issue #46. Key Findings | Method | Model | File F1 | Module F1 | Entity/Function F1 | | - | - | - | - | - | | LocAgent | Claude 3.5 | 0.4348 | 0.3402 | 0.1975 | | SWE-Fixer | Qwen2.5-7B/72B | 0.6917 | N/A | N/A | LocAgent (arXiv:2503.09089, GitHub: gersteinlab/LocAgent): • ✅ Provides full multi-level localization (file, module, function) • Has pre-computed results in

evaluation/loc_output/locagent/claude_3-5/loc_outputs.jsonl

• Can be evaluated at all three levels SWE-Fixer (arXiv:2501.05040, GitHub: InternLM/SWE-Fixer): • ❌ Does NOT make module-level or function-level predictions • Only performs file-level retrieval (BM25 + retriever model) • The editor generates patches directly without explicit module/function localization • Cannot be evaluated on module/function localization Changes Made 1.

scripts/evaluate_locagent.py

- Evaluates LocAgent results using our F1 metrics 2.

scripts/evaluate_swefixer.py

- Evaluates SWE-Fixer file-level results 3.

evaluation_results/locagent_evaluation.json

- Detailed per-instance results 4.

evaluation_results/swefixer_evaluation.json

- Detailed per-instance results 5.

evaluation_results/baseline_evaluation_summary.md

- Summary document Conclusion For module/function level baseline comparison, only LocAgent can be used since SWE-Fixer's architecture doesn't produce these predictions. SWE-Fixer's higher file-level F1 (0.6917 vs 0.4348) comes from evaluating the final patch output rather than intermediate retrieval. ------------------------- Would you like me to: 1. Push these changes to a new branch and create a PR? 2. Keep the changes local for now? 3. Any other workflow preference? _{View full conversation}

Summary I investigated the LocAgent and SWE-Fixer baselines for module-level and function-level localization evaluation. Key Findings LocAgent (arXiv:2503.09089, GitHub: gersteinlab/LocAgent): • ✅ Provides full multi-level localization (file, module, function) • Results on SWE-Bench Lite using our F1 metrics: ◦ File F1: 0.4348 ◦ Module F1: 0.3402 ◦ Entity F1: 0.1975 Why F1 differs from paper's Acc@k: The paper reports Acc@1=0.7774 for files, but our F1 is 0.4348. This is because: • LocAgent predicts 4.09 files on average while ground truth has 1.00 file on average • 93% of instances have more predictions than ground truth • Acc@k only checks if correct file is in top-k (recall-oriented), while F1 penalizes false positives • Example: predicting 4 files when 1 is correct gives Acc@1=1.0 but F1=0.4 SWE-Fixer (arXiv:2501.05040, GitHub: InternLM/SWE-Fixer): • ❌ Does NOT make module-level or function-level predictions • Only performs file-level retrieval (BM25 + retriever model) • File F1: 0.6917 (extracted from final patches, not intermediate retrieval) • Cannot be evaluated on module/function localization Files Added •

scripts/evaluate_locagent.py

- Evaluation script for LocAgent •

scripts/evaluate_swefixer.py

- Evaluation script for SWE-Fixer •

evaluation_results/

- Detailed JSON results and summary Conclusion For module/function-level localization baselines, only LocAgent is applicable. SWE-Fixer's architecture doesn't produce these predictions. ------------------------- Would you like me to: 1. Push these changes to a new branch and create a PR? 2. Keep the changes local for now? 3. Any other workflow preference? _{View full conversation}

Summary I've completed the evaluation of LocAgent and SWE-Fixer baseline methods for code localization. Here are the key findings: Why LocAgent's File F1 (0.4348) Doesn't Match the Paper The paper reports Acc@k metrics (e.g., Acc@1=0.7774), not F1 scores. The difference is: • Acc@k: 1 if ANY ground truth item is in top-k predictions (doesn't penalize over-prediction) • F1: Harmonic mean of precision and recall (penalizes both false positives and false negatives) LocAgent over-predicts files (avg 4.09 files per instance vs 1.0 GT), which: • Gives high Acc@k (correct file is usually in predictions) • Gives lower F1 (precision is hurt by extra predictions) Thresholding to Improve Scores Yes! Using only the top-1 prediction significantly improves F1: | Level | F1 (all) | F1@1 | Improvement | | - | - | - | - | | File | 0.4348 | 0.7733 | +78% | | Module | 0.3402 | 0.6453 | +90% | | Entity | 0.1975 | 0.4778 | +142% | Updated Evaluation Results LocAgent (Claude 3.5) on SWE-Bench Lite: | Level | F1 | F1@1 | Acc@1 | Acc@5 | |-------|-----|------|-------|-------| | File | 0.4348 | 0.7733 | 0.7733 | 0.9367 | | Module | 0.3402 | 0.6453 | 0.6599 | 0.8721 | | Entity | 0.1975 | 0.4778 | 0.5084 | 0.7980 | SWE-Fixer on SWE-Bench Lite: • File F1: 0.6917, Acc@1: 0.6900 • Module/Entity: N/A (SWE-Fixer only does file-level retrieval) Changes Pushed Branch:

openhands-workspace-hz6ixx85

Files added/modified: •

scripts/evaluate_locagent.py

- Now includes Acc@k, Recall@k, and F1@k metrics •

scripts/evaluate_swefixer.py

- Now includes Acc@k, Recall@k, and F1@k metrics •

evaluation_results/locagent_evaluation.json

- Detailed per-instance results •

evaluation_results/swefixer_evaluation.json

- Detailed per-instance results •

evaluation_results/baseline_evaluation_summary.md

- Updated summary document Would you like me to create a pull request for these changes? _{View full conversation}

Summary I've created the baseline comparison CSV table as requested and pushed it to the branch. CSV File Created:

evaluation_results/baseline_comparison.csv

System,File Acc@1,File Acc@5,File F1,Module F1,Function/Entity F1
SWE-Fixer,0.6900,0.6933,0.6917,N/A,N/A
LocAgent (return 1),0.7733,0.7733,0.7733,0.6453,0.4778
LocAgent (return 2),0.7733,0.8767,0.6078,0.5423,0.4406
LocAgent (return 3),0.7733,0.9133,0.5144,0.4505,0.3892
LocAgent (return all),0.7733,0.9367,0.4348,0.3402,0.1975

Key Findings 1. LocAgent (return 1) achieves the best F1 scores across all levels: ◦ File F1: 0.7733 (vs 0.6917 for SWE-Fixer) ◦ Module F1: 0.6453 ◦ Entity F1: 0.4778 2. SWE-Fixer only provides file-level predictions (no module/entity localization) 3. Trade-off: As LocAgent returns more predictions: ◦ Acc@5 increases (0.7733 → 0.9367) ◦ F1 decreases (0.7733 → 0.4348) due to more false positives Why LocAgent's F1 didn't match the paper The paper reports Acc@k metrics (0.77-0.94), not F1. LocAgent over-predicts files (avg 4.09 per instance vs 1.0 ground truth), which hurts F1 but not Acc@k. Using only the top-1 prediction gives F1 = Acc@1 = 0.7733. Branch All changes pushed to:

openhands-workspace-hz6ixx85

Would you like me to create a PR for these changes? _{View full conversation}