Skip to main content
AI Choice Engine
← Benchmark sources

swebench.com · benchmark source

SWE-bench

Repository-level GitHub issue resolution benchmarks. Verified (built by OpenAI with the SWE-bench team) is a human-filtered historical subset — saturated, with no official runs after February 2026.

Open swebench.com1 board on this publisher

Software engineering

SWE-bench Verified

A historical benchmark of real GitHub issues where an agent must modify a repository so tests and issue requirements are satisfied.

Why it matters

It gives coding agents a shared task language closer to real maintenance than a short code-generation prompt.

Use it when

You are reproducing historical repository-level results and can document the same harness assumptions; for current progress use DeepSWE 1.1, Terminal-Bench 4.0 or CursorBench 4.0.

The limitation

OpenAI reports contamination and grading flaws in SWE-bench Verified, so it should not be treated as current frontier-progress evidence. Harness, scaffold, test selection, patch filtering, and model effort also materially change the score.

Leaderboard snapshot

SWE-bench Verified results in a readable view

Use the tabs for the primary score and other published fields from this board. The source link remains authoritative for the live table.

Historical board. SWE-bench Verified is saturated and OpenAI cautions against using it as evidence of current frontier progress; the official board has no entries after February 2026, so 2026 frontier models (Claude Opus 5 / 5.5, Fable 5.x, GPT-5.6, GPT-6) have no official Verified run. For current coding signal use DeepSWE 1.1, Terminal-Bench 4.0 or CursorBench 4.0.
Official mini-SWE-agent bash-only submissions are ranked within each exact harness version; there is no cross-version winner. One vendor-reported claim appears separately without a rank. Compare only matching harness versions/configurations.

Leader among verified rows

Separate setup groups

Use the live source

Rows shown

47

Resolved

Source / snapshot date

2026-02-26

242 days old · not a live API feed

Open the live source before trusting rank order — especially when this snapshot predates recent model releases.

Score profile

Resolved · mini-SWE-agent 2.0.0 bash-only, 1 attempt

Higher is better in this view

Claude 4.5 Opus (high)
76.8%
Gemini 3 Flash (high)
75.8%
MiniMax M2.5 (high)
75.8%
Claude 4.6 Opus
75.6%
GLM 5 (high)
72.8%
GPT 5.2 (high)
72.8%
GPT 5.2 Codex
72.8%
Claude 4.5 Sonnet (high)
71.4%
Kimi K2.5 (high)
70.8%
DeepSeek V3.2 (high)
70%
Gemini 3 Pro
69.6%
Claude 4.5 Haiku (high)
66.6%
GPT 5 mini
56.2%

Local leader in this group: Claude 4.5 Opus (high) (76.8%)

Score profile

Resolved · mini-SWE-agent 1.16.0 bash-only, 1 attempt

Higher is better in this view

Claude 4.5 Opus (20251101) (medium)
74.4%
GPT 5.1 Codex (medium)
66%

Local leader in this group: Claude 4.5 Opus (20251101) (medium) (74.4%)

Score profile

Resolved · mini-SWE-agent 1.15.0 bash-only, 1 attempt

Higher is better in this view

Gemini 3 Pro Preview (2025-11-18)
74.2%
GPT 5.1 (2025-11-13) (medium)
66%

Local leader in this group: Gemini 3 Pro Preview (2025-11-18) (74.2%)

Score profile

Resolved · mini-SWE-agent 1.17.2 bash-only, 1 attempt

Higher is better in this view

GPT 5.2 (2025-12-11) (high)
71.8%
GPT 5.2 (2025-12-11)
69%
Kimi K2 Thinking
63.4%
Devstral Small (2512)
56.4%
Devstral (2512)
53.8%

Local leader in this group: GPT 5.2 (2025-12-11) (high) (71.8%)

Score profile

Resolved · mini-SWE-agent 1.13.3 bash-only, 1 attempt

Higher is better in this view

Claude 4.5 Sonnet (20250929)
70.6%

Local leader in this group: Claude 4.5 Sonnet (20250929) (70.6%)

Score profile

Resolved · mini-SWE-agent 1.0.0 bash-only, 1 attempt

Higher is better in this view

Claude 4 Opus (20250514)
67.6%
Claude 4 Sonnet (20250514)
64.93%
o3 (2025-04-16)
58.4%
Qwen3-Coder 480B/A35B Instruct
55.4%
Gemini 2.5 Pro (2025-05-06)
53.6%
o4-mini (2025-04-16)
45%
GPT 4.1 (2025-04-14)
39.58%
Gemini 2.5 Flash (2025-04-17)
28.73%
Qwen2.5-Coder 32B Instruct
9%

Local leader in this group: Claude 4 Opus (20250514) (67.6%)

Score profile

Resolved · mini-SWE-agent 1.7.0 bash-only, 1 attempt

Higher is better in this view

GPT 5 (2025-08-07) (medium)
65%
GPT 5 mini (2025-08-07) (medium)
59.8%
Kimi K2 Instruct
43.8%
GPT 5 nano (2025-08-07) (medium)
34.8%
gpt-oss-120b
26%

Local leader in this group: GPT 5 (2025-08-07) (medium) (65%)

Score profile

Resolved · mini-SWE-agent 1.17.0 bash-only, 1 attempt

Higher is better in this view

MiniMax M2
61%

Local leader in this group: MiniMax M2 (61%)

Score profile

Resolved · mini-SWE-agent 1.17.1 bash-only, 1 attempt

Higher is better in this view

DeepSeek V3.2 Reasoner
60%
GLM 4.6 (T=1)
55.4%

Local leader in this group: DeepSeek V3.2 Reasoner (60%)

Score profile

Resolved · mini-SWE-agent 1.9.1 bash-only, 1 attempt

Higher is better in this view

GLM 4.5 (2025-08-22)
54.2%

Local leader in this group: GLM 4.5 (2025-08-22) (54.2%)

Score profile

Resolved · mini-SWE-agent 0.0.0 bash-only, 1 attempt

Higher is better in this view

Claude 3.7 Sonnet (20250219)
52.8%
GPT 4.1 mini (2025-04-14)
23.94%
GPT 4o (2024-11-20)
21.62%
Llama 4 Maverick Instruct
21.04%
Gemini 2.0 Flash
13.52%
Llama 4 Scout Instruct
9.06%

Local leader in this group: Claude 3.7 Sonnet (20250219) (52.8%)

Separate vendor claims (1)

Sourced claims with different scaffolds or run conditions. They do not affect the official ranking, leader or charts.

  • Claude Haiku 4.5 (vendor-reported)73.3%

    Vendor-reported and not ranked against the official runs. The official bash-only run of the same model scores 66.6% — the different scaffold and run conditions make these results unsuitable for a controlled score comparison.

Official SWE-bench Verified bash-only submissions (mini-SWE-agent, single attempt; harness versions shown per row) captured 2026-10-01 from the SWE-bench team's public leaderboards.json; newest submission dated 2026-02-26. Vendor-reported Verified claims for newer models are not shown because they use private scaffolds and cannot be compared with these rows. SWE-Bench Pro (Scale AI) is a different benchmark and is not mixed in.

Open SWE-bench Verified leaderboard (official)

Evidence in this catalog

Where SWE-bench Verified fits

Closest verified examples we currently carry. A missing score is not a zero.

Model examples

  • Gemini 3 Flash (high)75.8%

    Published source observation captured 2026-10-01. Source-specific result; no independent rerun or universal capability verdict.

  • MiniMax M2.5 (high)75.8%

    Published source observation captured 2026-10-01. Source-specific result; no independent rerun or universal capability verdict.

  • GPT 5.2 (high)72.8%

    Published source observation captured 2026-10-01. Source-specific result; no independent rerun or universal capability verdict.

Harness examples

No directly comparable harness score is verified here yet.

Open this board on SWE-bench Verified leaderboard (historical)