What is SWE-bench?
Measures real-world software engineering ability by asking a model to resolve actual GitHub issues from open-source repositories, verified against the projects' own tests.
SWE-bench scores by model
1
Claude Opus 4.8Anthropic88.6% ↗
2
Claude Opus 4.7Anthropic87.6% ↗
3
Claude Sonnet 5Anthropic85.2% ↗
4
Claude Opus 4.6Anthropic80.8% ↗
6
Claude Sonnet 4.6Anthropic79.6% ↗
8
Gemini 3 Flash PreviewGoogle78% ↗
10
Mistral Medium 3.5Mistral AI77.6% ↗
11
Claude Sonnet 4.5Anthropic77.2% ↗
12
Qwen3.6 27BQwen77.2% ↗
13
Step 3.7 FlashStepFun76.5% ↗
14
Dola Seed 2.0 ProByteDance Seed76.5% ↗
15
Qwen3.5 397B-A17BQwen76.4% ↗
16
Qwen3.5 27BQwen75% ↗
18
Claude Opus 4.1Anthropic74.5% ↗
19
Step 3.5 FlashStepFun74.4% ↗
20
MAI-Thinking-1Microsoft73.5% ↗
21
Qwen3.6 35B-A3BQwen73.4% ↗
22
Claude Haiku 4.5Anthropic73.3% ↗
23
DeepSeek V3.2DeepSeek73.1% ↗
24
Claude Sonnet 4Anthropic72.7% ↗
25
Claude Opus 4Anthropic72.5% ↗
26
GPT-5 miniOpenAI71% ↗
27
Qwen3-Coder-NextQwen70.6% ↗
28
Solar Open 2 250B-A15BUpstage70.4% ↗
29
Claude 3.7 SonnetAnthropic70.3% ↗
30
NVIDIA Nemotron 3 Ultra 550B-A55B NVFP4NVIDIA69.7% ↗
32
Qwen3-Coder-PlusQwen69.6% ↗
34
North Mini CodeCohere67.6% ↗
35
gpt-oss-120bOpenAI62.4% ↗
36
gpt-oss-20bOpenAI60.7% ↗
37
NVIDIA Nemotron 3 Super 120B-A12BNVIDIA60.47% ↗
38
LongCat Flash ChatMeituan LongCat60.4% ↗
39
Gemini 2.5 ProGoogle59.6% ↗
40
GPT-5 nanoOpenAI54.7% ↗