deepswe.datacurve.ai · benchmark source
DeepSWE
Long-horizon engineering evaluation that publishes pass rate together with cost, output tokens, and agent steps.
Agentic coding
DeepSWE 1.1
A public long-horizon engineering evaluation that reports pass rate alongside cost, output tokens, and agent steps.
Why it matters
Cost per completed task exposes why a slightly weaker model can be the better production choice.
Use it when
You need to compare quality with the amount of work and money the agent spends to get there.
The limitation
Scores are tied to the published harness and task set; provider-specific tuning can skew comparisons.
Leaderboard snapshot
DeepSWE 1.1 — chart and full table
Same layout as DeepSWE’s public board: Cost / Output tokens / Agent steps chart with a v1.1 ↔ v1 toggle, then one full Pass@1 · Cost · Tokens · Steps table.
Local leader
GPT-6 Astra [xhigh]
74.1%
Rows shown
28
Best source-matched Pass@1 per model
Snapshot date
2026-10-01
Mirrored from deepswe.datacurve.ai
Better is toward the top-right (higher pass rate, lower cost). X-axis is reversed to match DeepSWE’s public chart. v1.1 uses average cost, output tokens and steps. Legacy v1 transcriptions are excluded here until an exact primary receipt is retained.
| # | Model | Pass@1 | Cost / task | Tokens / task | Steps / task |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra [xhigh] | 74.1% | $4.43 | 30k | 29 |
| 2 | Gemini 3.8 Flash [high] | 73.8% | $2.36 | 143k | 166 |
| 3 | Claude Opus 5 [max] | 73.6% | $11.84 | 118k | 99 |
| 4 | GPT-5.6 Sol [max] | 72.7% | $8.39 | 60k | 61 |
| 5 | Claude Fable 5 [xhigh] | 69.9% | $13.41 | 80k | 68 |
| 6 | GPT-5.6 Terra [max] | 69.6% | $4.95 | 72k | 76 |
| 7 | GLM 5.3 [max] | 69.0% | $3.99 | 80k | 124 |
| 8 | Kimi K3 [max] | 68.5% | $4.65 | 82k | 98 |
| 9 | Grok 4.6 [medium] | 67.5% | $3.45 | 50k | 70 |
| 10 | GPT-5.6 Luna [max] | 67.2% | $3.03 | 73k | 102 |
| 11 | GPT-5.5 [xhigh] | 67.0% | $7.23 | 46k | 82 |
| 12 | Gemini 3.7 Flash [medium] | 65.5% | $2.03 | 94k | 117 |
| 13 | GLM 5.3 Flash [max] | 63.4% | $0.48 | 73k | 123 |
| 14 | DeepSeek V4-Pro [max] | 62.8% | $0.24 | 106k | 155 |
| 15 | Claude Opus 4.8 [max] | 59.0% | $13.22 | 135k | 120 |
| 16 | Qwen3.8-Max [xhigh] | 57.5% | $3.73 | 95k | 111 |
| 17 | Muse Spark 1.2 [xhigh] | 54.9% | $3.70 | 99k | 101 |
| 18 | Claude Sonnet 5 [max] | 53.8% | $26.40 | 214k | 268 |
| 19 | Grok 4.5 [high] | 53.8% | $2.42 | 36k | 61 |
| 20 | DeepSeek V4-Flash [max] | 53.3% | $0.10 | 108k | 153 |
| 21 | Muse Spark 1.1 [xhigh] | 53.3% | $2.36 | 74k | 96 |
| 22 | GPT-5.4 [xhigh] | 51.8% | $5.65 | 71k | 70 |
| 23 | Gemini 3.6 Flash [high] | 46.7% | $4.42 | 96k | 117 |
| 24 | GLM 5.2 [max] | 43.8% | $3.92 | 78k | 129 |
| 25 | Gemini 3.5 Flash [high] | 36.1% | $3.45 | 76k | 105 |
| 26 | Kimi K2.7 Code | 30.5% | $2.82 | 59k | 149 |
| 27 | Claude Sonnet 4.6 [high] | 29.9% | $5.52 | 76k | 134 |
| 28 | Gemini 3.1 Pro [high] | 11.7% | $2.14 | 28k | 76 |
“Best per model” shows each model’s highest source-matched Pass@1 in this capture; switch to all effort levels to see every retained configuration. Efforts differ across models, so this view is not a comparison at matched effort. The source was generated September 22 and captured October 1. Small gaps may not be statistically meaningful — confirm on deepswe.datacurve.ai.
Evidence in this catalog
Where DeepSWE 1.1 fits
Closest verified examples we currently carry. A missing score is not a zero.
Model examples
- GPT-6 Astra [xhigh]74.1%
Published source observation captured 2026-10-01. Source-specific result; no independent rerun or universal capability verdict.
Published source observation captured 2026-10-01. Source-specific result; no independent rerun or universal capability verdict.
- Claude Opus 5 [max]73.6%
Published source observation captured 2026-10-01. Source-specific result; no independent rerun or universal capability verdict.
Harness examples
No directly comparable harness score is verified here yet.