Here's the answer to the Slack question **"How many models have we benchmarked in our lifetime, and what's the open vs closed breakdown?"**

---

Based on combining both sources — the older spreadsheet (`OpenHands Benchmark Performance.xlsx`, covering SWE-Bench Lite, SWE-Bench Verified, v1 SWE-Bench Verified, and GAIA evaluations from late 2024 through late 2025) and the current `openhands-index-results` repo (structured results from Jan 2026 onward across 5 benchmarks) — here's the tally:

## **69 unique models benchmarked in total**

| Category | Count | Share |
|---|---|---|
| **Open / open-weight** | **39** | **57%** |
| **Closed / proprietary** | **30** | **43%** |

### By provider:
| Provider | Models |
|---|---|
| OpenAI | 12 (GPT-4o, GPT-4o Mini, GPT-4.1, o1, o3-mini, o4-mini, GPT-5, GPT-5 Mini, GPT-5.2, GPT-5.2-Codex, GPT-5.4, GPT-OSS 120B) |
| Anthropic | 10 (Claude 3.5 Haiku/Sonnet, 3.7 Sonnet, Sonnet 4, Opus 4, 4.5 Haiku, Opus/Sonnet 4.5, Opus/Sonnet 4.6) |
| Google | 8 (Gemini 1.5 Flash/Pro, 2.0 Flash, 2.5 Pro, 3 Flash/Pro, 3.1 Pro, Gemma-3 27B) |
| Alibaba/Qwen | 8 (Qwen2.5-72B, Qwen2.5-Coder-32B, QwQ-32B, Qwen3-235B, Qwen3-Coder-480B, Qwen3-Coder-Next, Qwen3.5-Flash, Qwen3.6-Plus) |
| DeepSeek | 7 (V2/Chat, V3, V3 0324, R1, R1-0528, V3.1, V3.2 Reasoner) |
| Mistral | 5 (Codestral, Devstral Small 2505/2507, Devstral Medium 2507, Devstral 2512) |
| MiniMax | 4 (M2, M2.1, M2.5, 2.7) |
| Meta | 3 (Llama 3.1 70B, 3.1 405B, 3.3 70B) |
| Zhipu AI | 3 (GLM-4.5, GLM-4.7, GLM-5) |
| NVIDIA | 3 (Nemotron-3-Nano, Nemotron-3-Super, Trinity-Large-Thinking) |
| Moonshot AI | 2 (Kimi K2, Kimi-K2.5) |
| xAI | 1 (Grok Code Fast 1) |
| Other | 3 (OpenHands LM 32B, Quasar Alpha, Horizon Alpha) |

### Notes:
- Counted each distinct model checkpoint once (e.g., GPT-4o's two date variants = 1 model; DeepSeek V3.1 with/without thinking = 1 model)
- "Open" includes open-weight models even if only available via API (DeepSeek, Qwen, Kimi, MiniMax, etc.)
- "Closed" includes models where weights are not publicly available (OpenAI, Anthropic, Google, xAI, Qwen3.6-Plus via Dashscope-only)
- The repo currently tracks 26 models with structured multi-benchmark results; the spreadsheet contributed ~45 additional models from earlier evaluations
- This does NOT count alternative agent configurations (Claude Code, Codex, Gemini CLI, OpenHands Sub-agents) which test the same models with different agent frameworks