SWE-bench scores for the same model differ by 20 points depending on how they are measured

A 20-point gap for one model: what gpt-oss-120b shows about test conditions

SWE-bench Verified is widely used to compare AI models, but the same model can get very different scores depending on who runs the test. According to an analysis by issoh.co.jp, gpt-oss-120b scored 62.4 on SWE-bench Verified in OpenAI's model card and 62.0 in a comparison table published by Qwen, which is close to the same result. When NVIDIA measured the model for a comparison with Nemotron 3 Super, however, it scored 41.9, about 20 points lower.

The main reason for the gap is that NVIDIA had the model solve the tasks inside OpenHands, an agent execution framework. Even under the same "SWE-bench Verified" name, the harness affects the score. The harness is the execution environment, and it decides how much the model may explore the codebase and run tests. MMLU-Pro shows similar swings. OpenAI's model card gave an average of 81.3 across 14 non-English languages, while Qwen's comparison table listed 78.2, with no conditions stated alongside either number.

GPT-5 solves just 23% on SWE-bench Pro, far below its Verified results

The results from SWE-bench Pro, a new version that makes the evaluation itself harder, are even more striking. According to a report by Pat McGuinness, SWE-bench Pro has 1,865 long-horizon tasks of the kind that take professional software engineers hours or days to complete. The public dataset alone includes 731 tasks. GPT-5 ranked first on the benchmark with a solve rate of just 23%, and Claude Opus 4.1 came second. Both scores were far below the 60–70% range seen on SWE-bench Verified.

The difference comes from what each version contains. SWE-bench Verified is limited to 500 problems drawn from a pool of 2,294 and confirmed to be solvable. Pro includes more complex tasks that are closer to real-world work. Both carry the "SWE-bench" name, but they test problems of very different difficulty. That is a major pitfall for companies that look at these scores when deciding which models to adopt.

Claude 4, Qwen3.6 and Grok separated by narrow margins on Verified

The latest models from each company are competing closely on SWE-bench Verified. When it announced Claude 4, Anthropic said Claude Opus 4 scored 72.5% and Sonnet 4 scored 72.7%, and claimed the top spot on SWE-bench. Qwen3.6 Plus, by contrast, posted 78.8 on SWE-bench Verified, a sign of strength in repository-scale problem solving. In a comparison of Grok and OpenAI models, Grok 4.20 was reported at 75.0% and GPT-5.4 at 74.9% as of March 2026, a gap of just 0.1 points. Among newer small models, MiniCPM5-2B, a model with 2.6B parameters, has reportedly reached 46.4% on SWE-bench. Comparing such figures directly is risky unless the benchmark version and test conditions are stated.

MMLU scores also keep rising, with the race now above 90%

Scores are also climbing on MMLU, which measures knowledge and understanding. OpenAI announced that GPT-4.1 had set "a new standard of over 90% on MMLU reasoning." The same model scored 55% on SWE-Bench Verified, a big jump from the 33% of its predecessor, GPT-4o. Even so, it still trailed Google's Gemini 2.5 Pro, which scored 63.8% with agentic tool use, and Anthropic's Claude 3.7 Sonnet, which scored 62–63%. In comparisons of Grok and ChatGPT, Grok 3 scored about 84% on MMLU against 86.4% for GPT-5. On GPQA Diamond, Grok 3 in Think mode scored 84.6% and GPT-5 scored 85.7%. The race remains close on both benchmarks.

Experts warn that "a benchmark is not a verdict"

With so many competing numbers, an explainer from Fello AI says that "a benchmark is a single fixed test, not a verdict." On Humanity's Last Exam, for example, Claude Fable 5.1's result varies between 59.1%, 58.7% and 55.9% depending only on its reasoning-effort setting. The explainer describes SWE-bench as "three names for one family," because the "Verified" and "Pro" versions measure different things. OpenAI has also introduced GDPval, a new metric that measures tasks with high economic value, a sign that it accepts that older benchmarks such as MMLU and SWE-Bench are saturated and limited. When comparing scores, it will matter more and more to check who ran the test, which version was used and what execution environment it ran in.