
Essay Agent Benchmarks
Accuracy Metrics for Agentic Code Generation
Resolved rate conflates model capability with harness design and test quality in one number.
A coding agent gets a task, and its patch either passes a defined test suite or it doesn't; resolved rate is the fraction that pass, and it's the number every agentic coding system reports first. The mechanics are straightforward: an agent receives a task, generates a code change, and a test suite determines pass or fail with no human in the loop. SWE-bench built the template that most of the field still follows. It hands an agent a real GitHub issue along with a snapshot of the repository at the relevant commit, then asks the agent to understand the codebase, locate the right files, make the necessary changes, and pass a hidden set of tests. It draws on 2,294 real software engineering tasks pulled from 12 popular Python repositories, so the problems come from actual maintainer history, and not from constructed puzzles. Grading runs on two categories of unit tests, PASS_TO_PASS and FAIL_TO_PASS, and a patch only counts as resolved when it satisfies both for a given instance.
Earlier coding benchmarks struggled with a practical problem, and this design answers it. Variants built on the same foundation extended the approach further, and some are curated subsets meant to reduce noise and sharpen the signal. Nearly every new coding agent now gets evaluated against this template in some form, and labs, vendors, and buyers all track it as the baseline figure when comparing systems. The appeal is full automation, no human grader, and reproducibility given a fixed test suite. A single percentage obscures how much is actually being measured underneath it, conflating model capability with harness design, test quality, and task difficulty into one number.
What a resolved rate number conflates
A resolved-rate score looks like a clean measurement of model capability, but attributing it cleanly to the model alone is not possible. An agentic coding system is not a model running in isolation. Changing any one of those components can move the resolved rate by an amount comparable to the gap between two entirely different model generations.
Harness-Bench, published in May 2026, demonstrated this directly. The paper's conclusion was blunt: capability claims should attach to a model-and-harness pairing, not to a model alone. A position paper published in June 2026 pushed the same argument further, and it named three specific symptoms of benchmark misalignment in agentic software engineering. The first is conflating the model with the rest of the harness when reporting a score. The third is the absence of any signal below the end-to-end level, so a failure gives no indication of which component caused it.
The practical consequence for anyone reading a leaderboard is immediate. When two vendors report different resolved rates for what is nominally the same base model, the harness wrapped around that model is usually the explanation, not some underlying difference in capability. A resolved-rate number, read in isolation, tells you almost nothing about whether you're looking at a better model or a better-engineered scaffold around a comparable one.
Test quality and contamination erode the signal inside resolved rate
Harness variance is a problem with attribution. A second, deeper problem sits in the measurement apparatus itself: if the tests grading a patch are wrong, or if the model has already seen the answer during training, resolved rate stops measuring capability or generalization and starts measuring something else.
Test quality turns out to be a real and quantified failure mode, not a hypothetical one. The benchmark's manually written tests often failed to catch an incorrect patch, so one ACL 2025 paper built an LLM-driven test generator to fix that. In other words, a meaningful share of the patches credited as successes on two of the most widely cited benchmark variants were not actually correct fixes, and the error was large enough to reorder which systems looked best.
Contamination compounds the problem. OpenAI's response to this is the clearest signal of how seriously the field has started taking the issue. A manual audit OpenAI ran on a set of its own o3 failures found that a majority of them traced back to flaws in the test suite.
None of this is unique to the original benchmark's construction. A 2026 audit looked at a successor benchmark built to fix earlier generations' weaknesses, and it still found dataset quality issues in roughly a third of cases, including test cases that were either broken outright or so strict they penalized valid fixes. The lesson generalizes: building a harder or newer benchmark does not automatically escape the same test-quality failures that undermined the benchmark it replaced. Procurement teams and research labs use these scores to justify model selection or funding decisions, but they may be measuring neither real capability nor genuine progress, and the more these audits accumulate, the harder that gap is to ignore.
What happens to resolved rate when tasks get harder
Resolved rate also has flawed tests and contaminated training data as floor problems, but it runs into a ceiling problem too once the tasks stop resembling single-issue bug fixes. A strong score on the standard benchmark turns out to predict very little about how an agent performs on work that more closely resembles the scope of real engineering, and the gap is large enough to describe as a performance cliff.
FeatureBench makes the point concretely. One model that achieves a strong resolved rate on the standard benchmark succeeds on only 11.0% of a harder benchmark's tasks, which are built around sustained feature work. It is a collapse from a benchmark that rewards locating and patching a bounded defect to one that requires sustained, multi-step feature construction, showing that the capability the standard benchmark rewards does not transfer to the kind of work most engineering teams actually need agents to do.
RoadmapBench, published in May 2026, pushes in a similar direction: its long-horizon tasks are grounded in real open-source version upgrades across multiple repositories and programming languages, and each one requires changes across a large number of lines spread over many files. DeepSWE takes yet another approach to avoiding memorization specifically: it is built around 113 original long-horizon engineering tasks written from scratch across TypeScript, Go, Python, JavaScript, and Rust, each running in an isolated environment with program-based verifiers, lowering the odds that a model's apparent success reflects reconstruction of a memorized solution.
The widening gap between scores on the standard benchmark and performance on these harder, longer-horizon benchmarks is evidence that the resolved-rate signal coming out of standard benchmarks measures something considerably narrower than agentic capability as practitioners actually need it, not a calibration detail to be smoothed over with a correction factor.
A resolved-rate score looks like a clean measurement of model capability, but attributing it cleanly to the model alone is not possible.
What resolved rate misses: functional correctness versus review-ready code
Even on tasks where resolved rate produces a clean, uncontaminated, well-tested signal, it still measures only one dimension of what makes a code change acceptable to a real engineering team. Passing a test suite is a necessary condition for a usable patch. It is not sufficient.
Treating an accepted diff as the unit of success implies a specific, demanding standard: the final state of the workspace has to pass a hidden behavioral verifier, keep every pre-existing check green, preserve interfaces that are supposed to remain untouched, and avoid any prohibited changes to files outside the task's scope. Resolved rate, as typically reported, checks only the first of these conditions. It says nothing about whether the patch touched files it shouldn't have, whether it introduced a change that happens to pass the test suite while violating an interface contract, or whether a human engineer would need to intervene before the change could be merged into production. If a reviewer has to edit the generated code before it ships, the autonomous run failed at its actual job, regardless of how trivial that final edit turns out to be.
This distinction matters because the unit of comparison practitioners care about is not "did the agent eventually produce passing code" but "can this output be merged as delivered." A task that consumes agent compute time and then requires a substantial human review and repair cycle before it's mergeable is a fundamentally different outcome than a run that merges cleanly on the first attempt, even if both runs get counted identically under a standard resolved-rate calculation. The metric cannot express that difference, because it was built to answer a narrower question than the one engineering teams are actually asking.
The operational metrics that capture what resolved rate leaves out
Closing the gap between "passes tests" and "ready to merge" requires a layer of operational metrics that sit alongside resolved rate. Taken together, these metrics describe how an agent behaves across the full lifecycle of a task, not just whether it eventually produced a passing patch.
First-pass success tracks whether the agent produces an acceptable diff on its very first attempt, with no retry. Retry rate makes that masking visible: a system with a low resolved rate and a high retry rate is failing in a fundamentally different way than a system with the same resolved rate and a single clean failure per task, and conflating the two under one number erases a distinction practitioners need to see.
Repeated-run reliability addresses a related question: whether running the same task multiple times converges on consistently accepted solutions. An agent that succeeds on a given share of independent runs by behaving inconsistently, succeeding sometimes and failing other times on the identical task, is not interchangeable with an agent that succeeds on that same share by solving one fixed subset of tasks reliably and failing the rest every time. The two systems might report the same aggregate resolved rate, but they are very different tools to build around.
Review burden measures the human effort required after an agent produces output, typically in terms of the code edits a reviewer has to make before the change can merge. This metric directly resolves the ambiguity the previous section raised between a diff that merely passes tests and one that's genuinely mergeable. Elapsed time and the number of agent steps, meaning wall-clock duration and the count of tool calls or loop iterations a task consumed, matter for a related reason: a task that takes several minutes of agent compute followed by a lengthy human review session is not equivalent in practice to a task that completes in comparable wall time and merges without further intervention.
Cost per accepted task closes out this layer by dividing total compute and API spend by the number of tasks actually accepted, not merely completed. Frontier model pricing now diverges meaningfully even among models that report statistically indistinguishable resolved rates, so this metric is becoming a first-class consideration; two systems with near-identical accuracy can represent very different economic propositions once you factor in cost per accepted task.
Component-level metrics and the emerging benchmarks designed around them
The operational metrics above describe outcomes: what happened to a given task, how long it took, and what it cost. They still don't resolve the question of attribution raised earlier: when a task fails, which component of the agentic system was responsible. End-to-end resolved rate provides no way to localize a failure to the model's reasoning, the harness's tool orchestration, or the context pipeline's handling of a large codebase, so a new generation of benchmarks has started targeting those components directly.
PEEK, published by Gu and colleagues in 2026, evaluates a long-context agent system on long-context aggregation and context-learning tasks, and not on end-to-end completion. This isolates the model's ability to process and use codebase context on its own, apart from whatever harness wraps around it, addressing what Harness-Bench and the June 2026 position paper identified as the attribution gap. DecisionBench, from Gao and colleagues in 2026, takes aim at a different component: it evaluates how well an agent delegates sub-tasks across a pool of available models, targeting the orchestration layer of a multi-agent system rather than any single model's raw generation ability.
ProgramBench, published by Yang and colleagues in 2026, changes the unit of evaluation itself. It shifts the grading target from patch-based issue resolution to whole-program reconstruction, graded through agent-generated behavioral tests. The June 2026 position paper's second symptom was that grading against one reference solution penalizes equally valid alternative implementations, and grading against behavior instead addresses that, so it may prove more robust to the kind of harness variance that makes a single end-to-end resolved rate so hard to interpret on its own.
Taken as a whole, the direction these benchmarks point toward is a layered evaluation picture. Resolved rate still has a place in that picture: it's fast, automated, and grounded in real repositories, and that combination is why it took hold in the first place. You need operational metrics and component-level benchmarks to answer the questions resolved rate was never built to ask.
Notes
- FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
Provided the FeatureBench results showing a strong SWE-bench performer succeeds on only 11.0% of harder feature-development tasks.
- Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
Provided the June 2026 position paper's three symptoms of benchmark misalignment, including the attribution gap and the penalty for alternative valid implementations, and named PEEK and DecisionBench as component-level responses.
- UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
Provided the UTBoost finding that manually written benchmark tests failed to catch incorrect patches, leading to an LLM-driven test generator to correct the signal.
- SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
Provided the 2026 audit of SWE-Bench Pro Verified finding dataset quality issues in roughly a third of cases, including broken or overly strict test cases.


