deepswe.datacurve.ai · benchmark source
DeepSWE
Long-horizon engineering evaluation that publishes pass rate together with cost, output tokens, and agent steps.
Agentic coding
DeepSWE 1.1
A public long-horizon engineering evaluation that reports pass rate alongside cost, output tokens, and agent steps.
Why it matters
Cost per completed task exposes why a slightly weaker model can be the better production choice.
Use it when
You need to compare quality with the amount of work and money the agent spends to get there.
The limitation
Scores are tied to the published harness and task set; provider-specific tuning can skew comparisons.
Leaderboard snapshot
DeepSWE 1.1 — chart and full table
Same layout as DeepSWE’s public board: Cost / Output tokens / Agent steps chart with a v1.1 ↔ v1 toggle, then one full Pass@1 · Cost · Tokens · Steps table.
Local leader
Claude Opus 5 [max]
74%
Rows shown
24
Highest published reasoning effort per model (not best Pass@1)
Snapshot date
2026-08-13
Mirrored from deepswe.datacurve.ai
Better is toward the top-right (higher pass rate, lower cost). X-axis is reversed to match DeepSWE’s public chart. v1.1 uses average cost / tokens / steps; v1 uses published medians.
| # | Model | Pass@1 | Cost / task | Tokens / task | Steps / task |
|---|---|---|---|---|---|
| 1 | Claude Opus 5 [max] | 74% | $11.84 | 118k | 99 |
| 2 | GPT-5.6 Sol [max] | 73% | $8.39 | 60k | 61 |
| 3 | Claude Fable 5 [max] | 70% | $21.63 | 119k | 88 |
| 4 | GPT-5.6 Terra [max] | 70% | $4.95 | 72k | 76 |
| 5 | Kimi K3 [max] | 69% | $4.65 | 82k | 98 |
| 6 | GPT-5.6 Luna [max] | 67% | $3.03 | 73k | 102 |
| 7 | GPT-5.5 [xhigh] | 67% | $7.23 | 46k | 82 |
| 8 | Grok 4.6 [xhigh] | 67% | $5.50 | 71k | 87 |
| 9 | Gemini 3.7 Flash [high] | 65% | $2.18 | 107k | 125 |
| 10 | DeepSeek V4-Pro [max] | 63% | $0.24 | 106k | 155 |
| 11 | Claude Opus 4.8 [max] | 59% | $13.22 | 135k | 120 |
| 12 | Qwen3.8-Max [xhigh] | 58% | $3.73 | 95k | 111 |
| 13 | Muse Spark 1.2 [xhigh] | 55% | $3.70 | 99k | 101 |
| 14 | Claude Sonnet 5 [max] | 54% | $26.40 | 214k | 268 |
| 15 | Grok 4.5 [high] | 54% | $2.42 | 36k | 61 |
| 16 | DeepSeek V4-Flash [max] | 53% | $0.10 | 108k | 153 |
| 17 | Muse Spark 1.1 [xhigh] | 53% | $2.36 | 74k | 96 |
| 18 | GPT-5.4 [xhigh] | 52% | $5.65 | 71k | 70 |
| 19 | Gemini 3.6 Flash [high] | 47% | $4.42 | 96k | 117 |
| 20 | GLM 5.2 [max] | 44% | $3.92 | 78k | 129 |
| 21 | Gemini 3.5 Flash [high] | 36% | $3.45 | 76k | 105 |
| 22 | Kimi K2.7 Code | 31% | $2.82 | 59k | 149 |
| 23 | Claude Sonnet 4.6 [high] | 30% | $5.52 | 76k | 134 |
| 24 | Gemini 3.1 Pro [high] | 12% | $2.14 | 28k | 76 |
DeepSWE “Best” picks the highest published reasoning effort per model (not the highest pass rate). Small gaps may not be statistically meaningful — confirm on deepswe.datacurve.ai.
Evidence in this catalog
Where DeepSWE 1.1 fits
Closest verified examples we currently carry. A missing score is not a zero.
Model examples
- Claude Opus 574% pass@1 · $11.84
Catalogued DeepSWE result mirrored from deepswe.datacurve.ai.
- GPT-5.6 Sol73% pass@1 · $8.39
Catalogued DeepSWE result mirrored from deepswe.datacurve.ai.
- DeepSeek V4-Flash53% pass@1 · $0.10
Public DeepSWE v1.1 snapshot with a materially lower recorded task cost.
Harness examples
No directly comparable harness score is verified here yet.