
Essay Agent Benchmarks
SWE-Bench Verified Results Interpreted for Practitioners
Understanding what SWE-bench Verified scores actually measure.
SWE-bench Verified results measure how well a model works within a particular agent system, which includes more than the model’s capability on its own. For practitioners choosing a coding tool, the central task is to separate that system’s contribution from training contamination, benchmark scope, and automated grading. A leaderboard score becomes useful only when those conditions are clear and match the decision being made.
What SWE-bench Verified actually measures — and what it does not
SWE-bench draws real GitHub issues from popular open-source Python repositories, including Django, Flask, and scikit-learn. Researchers primarily from Princeton University created the benchmark, with additional contributors from Stanford University and the University of Chicago. Unlike HumanEval, which tests isolated function completion, SWE-bench tests a broader workflow. The agent must read an issue, navigate a real codebase, write a fix, and verify that it holds.
SWE-bench’s published specification defines Verified as a curated subset of exactly 500 tasks, created in collaboration with OpenAI. Software engineers reviewed each instance for a clear problem description, correct test patches, and solvability with the available information. The model is tested inside an isolated Docker container, where it cannot see the test cases themselves. The acceptance rule is binary. A generated patch must make tests that failed before pass while keeping existing tests passing.
That rule defines the result, but not everything that produces it. The model works through an agent scaffold that manages context and tool calls as it attempts the repository task. Its score therefore reflects both model capability and the system through which that capability is applied. The unit of measurement is the configured system.
Why one benchmark name now produces several incomparable numbers
Claude Opus 4.5 provides a direct example of scaffold sensitivity. Three agent systems ran the same model against SWE-bench Pro, and their scores were more than five points apart. Differences in how the agents managed context and tool calls accounted for the spread. Scale AI’s analysis also found that harness choices alone led to large swings in scores.
That finding complicates comparisons presented under the broader SWE-bench label. At minimum, readers encounter four distinct measurements, namely vendor self-reported Verified results, Pro results on vendor scaffolds, Pro results on Scale’s standardized SEAL harness for the public set, and Pro results on its private commercial subset. Although Pro and Verified are separate benchmarks, loose leaderboard comparisons can obscure that distinction as well as the differences within Pro.
A vendor’s best scaffold measures what its configured system can achieve on its infrastructure. A standardized harness puts models in the same setting so they can be compared, while a private subset changes which tasks they attempt. Those differences alter the meaning of the result before any question of model quality arises. An unstated harness leaves the comparison unresolved. When the same model can move by more than five points, a smaller lead cannot establish superiority without comparable evaluation conditions.
How data contamination inflated Verified scores and forced a benchmark reckoning
Scaffolding is not the only variable hidden behind a Verified score. OpenAI’s Frontier Evals team documented models reproducing verbatim gold patches or problem statements from a task ID alone. OpenAI’s public statement described contamination across all tested frontier models. This is more specific than familiarity with programming patterns. It indicates exposure to evaluation material, so successful completion can partly reflect memorization.
OpenAI announced in February 2026 that it would stop reporting SWE-bench Verified scores, citing contamination and flawed tests, and shift to SWE-bench Pro as its primary code evaluation. That withdrawal is the clearest institutional signal of the benchmark’s structural problem. It does not establish that the tested models lack coding capability. Rather, it weakens the inference that their Verified results demonstrate capability on unseen work. Vals.ai removed Verified from its index in August 2026, citing saturation.
An independent run can establish what a system scored under the evaluator’s conditions, but it cannot turn previously encountered tasks into unfamiliar ones. Nor does it remove scaffold dependence or extend a Python-only benchmark to other languages. A reproducible score can still be a poor estimate of performance outside the benchmark.
The consistent gap between Verified scores and SWE-bench Pro scores
Scale’s contamination-resistant benchmark uses real-world software tasks across dozens of professional repositories and scores performance using Pass@1. Its private commercial subset includes codebases legally inaccessible to model trainers. Public and private results still need to be kept separate.
Across frontier models, high Verified performance gives way to roughly sixty percent of Pro tasks solved under standardized scaffolding. Gemini 3.1 Pro shows how large that drop can be, with a score of 80.6% on Verified that falls sharply on Scale’s standardized public set. A difference of that kind can change a procurement decision. The relevant finding is the change in performance under different conditions.
Pro also has its own three-number problem. The top standardized public result, the top vendor-aggregate result, and the top private commercial result can each be presented as the best Pro score, despite differences in scaffolding and data splits. For practitioners, the useful calibration is a model’s Verified result paired with its Pro result under standardized scaffolding. A large gap is diagnostic of sensitivity to the evaluation conditions, though it does not isolate how much comes from scaffolding, contamination, or the changed task set.
SWE-bench’s published specification defines Verified as a curated subset of exactly 500 tasks, created in collaboration with OpenAI.
Why test-passing patches still fail human code review at high rates
METR published a study in March 2026 in which four active maintainers from scikit-learn, Sphinx, and pytest reviewed 296 AI-generated pull requests that had passed the automated grader. Roughly half would not have been merged. Maintainer judgments therefore placed a substantial limit on what a passing result demonstrated about a patch’s readiness for a real repository.
The rejections were not confined to stylistic preferences. Maintainers identified core functionality failures missed by the tests, breakage in other code outside test coverage, and code quality or convention problems. The grader establishes whether a patch satisfies the tested conditions, while maintainers judge whether it belongs in the codebase. Interpreting a sixty percent Verified score as resolution of sixty percent of real issues skips that distinction.
METR’s authors also identified an important constraint in their study. Agents could not iterate on their solutions in response to feedback as human developers would, so the results do not establish a fundamental capability limit. That qualification leaves the benchmark interpretation intact. The quoted score does not measure a feedback-driven review process, and any gains from that process cannot be assumed from the single-pass result. Maintainer-assessed merge quality is improving more slowly than automated grading performance, so higher benchmark scores alone do not show that the review gap is closing quickly.
Where Verified remains a valid signal and where it structurally fails
Verified retains value as a coarse filter between capability tiers. A model that cannot clear a basic threshold is unlikely to perform well on harder repository tasks, and the benchmark can distinguish the bottom of the field from the middle. Its strongest use is this broad separation. Fine-grained ranking within the top tier is less defensible when neighboring scores differ by less than the variance attributable to the harness.
Language scope sets a separate, firm boundary. Verified is Python-only, so it provides no direct evidence of performance in the other languages used by a polyglot team. The SWE-bench Multilingual paper describes 300 tasks across JavaScript, TypeScript, Rust, Ruby, Go, C, C++, PHP, and Java. A strong Python repository score cannot substitute for evidence in the languages a team actually maintains.
Verified uses public open-source repositories, where training contamination is structurally possible, while Pro’s private commercial subset is a closer proxy for proprietary work because its codebases are legally inaccessible to trainers. SWE-bench-Live addresses a different exposure problem. Its paper describes evaluation restricted to instances that post-date a model’s training cutoff, targeting temporal contamination rather than language coverage or human review quality.
A practical framework for reading any SWE-bench score before acting on it
The practical response is to apply five diagnostic questions before using a score to choose a tool. The first is whether the scaffold is stated. Reject a score as comparison evidence if its harness is undisclosed, because model identity alone does not identify the system being measured. Look for how context-management and tool-use were set up for the result.
Second, check whether the result is Verified or Pro and whether a corresponding score exists on the other benchmark. Treat Verified as a tier-filter, and require a Pro result under standardized scaffolding before comparing models for procurement. Inspect the gap. A large difference can signal scaffold sensitivity, contamination, or both, but the comparison also needs the Pro data split stated. A vendor-scaffold result and a standardized public-set result cannot serve as substitutes simply because both use Pro tasks.
Third, check whether the result is self-reported or independently run. Prefer the SWE-bench team’s own runs, Scale SEAL public leaderboard entries, or Vals.ai results where available. As of June 2026, nearly all leaderboard entries were self-reported. Treat vendor-submitted numbers as directional evidence. Independent evaluation strengthens confidence in the reported outcome, but the task set and harness still determine what that outcome means.
For multilingual work, use SWE-bench Multilingual or an appropriate Pro commercial-subset result as a closer proxy than Verified. For proprietary repositories, prefer the private commercial subset of Pro, whose rules on access are designed to keep those codebases out of model training. Scope comes before a small numerical advantage.
Account for METR’s finding that roughly half of test-passing patches would not survive maintainer review, without treating that sample as a universal conversion formula for every model or workflow. Keep test acceptance separate from merge readiness. When cost matters, also use score-per-dollar figures on leaderboards such as Morph’s SWE-bench Pro leaderboard, because a solve rate alone does not show how much each solved task costs.
CodeAnt AI complements benchmark-based selection by evaluating models against real repository tasks, including PR-level code review and security scanning. That gives teams a direct signal on their own codebases rather than one confined to Verified’s public Python repositories. This is a distinct role from leaderboard comparison.


