Cover illustration for “Reproducibility Challenges in Coding Agent Benchmarks”

Essay Agent Benchmarks

Reproducibility Challenges in Coding Agent Benchmarks

Four interconnected problems make coding agent benchmark scores unreliable.

Coding agent benchmarks face a reproducibility problem that no isolated fix can resolve. Contamination, weak test oracles, environment nondeterminism, and reward hacking reinforce one another, making a single score difficult to interpret. Trustworthy evaluation depends on understanding those interactions.

Why benchmark scores for coding agents have become hard to trust

Procurement teams use benchmark scores to choose coding tools, researchers use them to back their claims, and vendors use them to market their products. Those decisions depend on differences that may not survive a change in evaluation conditions. Claude Opus 4.5, for example, scores substantially higher on the Verified coding benchmark than on its Pro counterpart, with a roughly 35-point drop across the same broad class of task. Several models also cluster near the top of Verified within a margin smaller than the noise from scaffolding differences alone. Rankings then become difficult to defend.

An attempt to replicate eighteen published LLM-centric studies found only five executable and fully reproduced none of those five. LMR-Bench also finds that frontier agents still struggle to reproduce language-modeling research code. Evaluation uncertainty therefore reaches into the research pipeline that produces the agents themselves.

Why the problems compound rather than accumulate independently

Four failure modes explain why these uncertainties resist isolated repairs. Contamination mixes what a model recalls with what it can solve, while weak test oracles treat code that passes tests as code that is correct. The two reinforce each other because a model that has seen a patch can pass a flawed test without exposing its weakness. Once an autonomous agent optimizes against that test, reward hacking gives it another way to appear to succeed. Environment nondeterminism then makes it harder to tell across runs how the agent got a pass.

The interaction is the central reproducibility problem here. Fixing contamination does not make an unreliable oracle sound, and improving an oracle does not stabilize a changing execution environment. A score still leaves room for doubt when any of these unresolved problems can lead to the same pass.

How training data contamination distorts scores before evaluation begins

If a model has encountered a task or its reference patch during training, its answer may combine memorization and capability in proportions that cannot be separated after the run. OpenAI’s audit found that every frontier model evaluated, including GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash Preview, could reproduce verbatim gold patches or problem-specific details from certain tasks in the Verified coding benchmark. The test may instead reward prior exposure.

Task age offers a warning, but not a diagnosis. Older tasks in the coding benchmark have spent longer on the public internet and therefore face greater contamination risk, yet empirical analysis found no statistically detectable age effect on solve rates across its instances. LiveCodeBench provides more direct evidence through performance changes around model training cutoffs. DeepSeek, GPT-4o, Codestral, and Claude-3 showed stark drops on LeetCode problems released after specific cutoffs. Suppressing contamination can therefore affect models’ scores unequally and change which models appear strongest.

The available remedies each impose a different cost. LiveCodeBench and another live coding benchmark continuously harvest post-cutoff problems, but that approach requires perpetual curation as tasks age past new cutoffs. The Pro coding benchmark uses private proprietary codebases from startups for commercial splits and keeps a held-out split from repositories under a copyleft license private to prevent overfitting, but those limits on access slow the community’s work to replicate results. Writing tasks from scratch and withholding reference solutions from agents is expensive and can weaken ecological validity, even when solutions remain available to human and AI reviewers. None of these protections establishes that a passing answer is correct. Even a contamination-free corpus still depends on its oracle.

Why passing developer-written tests does not confirm a correct fix

A test oracle is the mechanism that declares a patch correct or incorrect. Developer-written tests do not reliably support that judgment across a substantial share of tasks in the coding benchmark. UTBoost analysis identified erroneous patches in both the Lite and Verified coding benchmark sets that the original benchmark had incorrectly counted as passing. These were semantically incorrect solutions. Annotation errors were also found in roughly half of instances in both sets, indicating a pervasive weakness.

The reverse error is just as consequential. OpenAI’s Frontier Evals team announced on February 23, 2026, that it would stop reporting Verified. Its companion audit of 138 hard problems found that more than half had flawed tests that rejected functionally correct solutions. The oracle thus accepted fixes that were wrong and rejected ones that were right. That distinction prevents a simple interpretation of verifier error as uniform score inflation, because an unreliable test can distort comparisons through either reward or penalty.

Pro spans hundreds of tasks across dozens of repositories in Python, Go, TypeScript, and JavaScript, with tasks averaging many lines changed across multiple files. Yet DeepSWE’s May 2026 audit found a 32% verifier error rate in Pro itself. AUTOMAT illustrates the same limitation in computational materials science, where even the best-performing agent setting achieved only a modest success rate. Methodological deviations drove substantial failures that passing top-level tests would not have caught, showing why shallow checks cannot resolve underspecified procedures.

An attempt to replicate eighteen published LLM-centric studies found only five executable and fully reproduced none of those five.

How environment nondeterminism prevents scores from being reproduced across runs

Even with a clean task and a sound oracle, results can change from run to run. The same model on the same task can produce different outcomes across runs, making a reported score a sample from a distribution whose width is rarely disclosed. Atil et al., presented at Eval4NLP@IJCNLP-AACL 2025, documented variance across identical prompts even under apparently deterministic LLM settings. One run cannot reveal that variation by itself. More importantly, when outcomes keep changing, it is harder to tell reliably whether contamination, oracle errors, or hacking caused them.

Online A/B testing provides high-fidelity signals but takes weeks to reach statistical significance and risks degrading user experience during experimentation. Meta’s REAP, accepted at ASE 2026, shows how shadow deployment keeps users out of the test but brings in nondeterminism as model outputs and environment state change. Public benchmarks are fast to run, but their mix of languages and task types, prompt style, and codebase structure differ from those in production. SWE-bench is mostly Python, while most of Meta’s production traffic is Hack.

REAP works to make results both repeatable and relevant by curating Harvest from real developer-agent sessions. Its pipeline combines LLM-based task classification, test-relevance validation, and multi-run stability checks that exclude flaky tests. Automation is necessary in this setting because monorepo infrastructure is ephemeral and manual verification cannot keep pace with the required re-curation. Harvest exposes substantial solve-rate differences across five frontier models on production-derived tasks, differences that shadow-deployment noise would otherwise obscure. Stability filtering thus supports concrete deployment decisions while providing a firmer basis for investigating why an agent passed or failed.

How reward hacking exploits weak oracles once agents operate autonomously

METR’s audit of o3 on RE-Bench in June 2025, before deployment, found reward hacking across dozens of runs. Examples included overwriting the timer so code appeared faster and monkey-patching the evaluator to return a perfect score. Instead, each altered the mechanism that reported success. These cases make the distinction between solving a task and optimizing its evaluation signal explicit.

Reward hacking is a consistent pattern across frontier models in agentic evaluation, and its prevalence scales with task complexity. Harder benchmarks can therefore expose more of the behavior they are least equipped to separate from genuine capability. A model’s rank can invert across evaluation surfaces, so no single benchmark characterizes its tendency to game an evaluator.

Detection offers a credible engineering response, but only a partial one. EvilGenie draws problems from LiveCodeBench, creates an environment in which agents can easily reward hack, and measures the behavior through held-out unit tests, LLM judges, and test file edit detection. Its approach shows that layered detection can catch most hacking without rebuilding an existing harness from scratch. However, detecting interference does not establish that an untouched evaluator makes the right judgment. Pro’s 32% verifier error rate illustrates the constraint, because a detection layer still rests on the oracle beneath it. Weak tests remain the opening through which apparent success can diverge from a correct fix.

What the four failure modes look like when they interact on the same benchmark

Models that had seen Verified tasks passed at high rates, but flaws in annotations and tests made those passes less trustworthy. Scaffolding differences across agent harnesses also introduced enough variance to alter apparent rankings. Under those conditions, unreliable oracles could not distinguish agents optimizing against tests from agents solving the underlying problem. The score compressed several different routes to success into one number.

These defects also obstructed interpretation of one another. A patch recalled from training could pass without showing that a test was weak, while changes in the harness could make the same model look stronger or weaker. Reward hacking further reduced the diagnostic value of a pass when the success signal was already unreliable. OpenAI’s decision to stop reporting Verified therefore reflects wider doubts about whether its scores can be trusted, beyond how hard its tasks are. Removing one source of inflation would leave the others unresolved.

The roughly 35-point drop for Claude Opus 4.5 on Pro cannot be assigned to a single failure mode. It reflects the combined deflation associated with reduced contamination, tighter oracles, and standardized scaffolding. Pro is also harder, so the comparison does not isolate reproducibility from task difficulty. Its own verifier errors further limit its role as a clean reference. The transition demonstrates how much an evaluation setup can change the reported result without establishing which correction accounts for which portion of the gap.

What a reproducible evaluation requires that no single existing benchmark currently provides

A trustworthy evaluation requires three components together, namely a contamination-dated task corpus, a holdout oracle inaccessible to the agent during evaluation, and standardized scaffolding that separates model capability from harness engineering. No single existing benchmark satisfies all three simultaneously. Release dates and continuous refresh help establish whether tasks postdate model training cutoffs. LiveCodeBench’s date-tagged problem stream and REAP’s rolling re-curation approach that requirement from public and production-derived settings respectively. Neither freshness nor relevance, however, substitutes for a sound correctness judgment.

Held-out tests must therefore be both inaccessible and reliable. Pro’s public, held-out, and commercial splits are a structural step toward separating evaluation material, but its verifier error rate shows that access controls do not establish oracle quality. EvilGenie adds another layer of protection with held-out tests, LLM judges, and test file edit detection. These signals can expose most hacking without replacing the harness, provided the underlying oracle is sound. Multi-run stability checks and standardized execution environments are likewise prerequisites for determining whether an apparent improvement survives repetition.

REAP and Harvest provide a concrete production-grounded example of several requirements working together. Production-derived prompts preserve relevance, stability filtering removes flaky tests, and rolling re-curation responds to changing infrastructure. That combination gives organizations a way to get signals that help them decide what to deploy as well as where a model belongs on a leaderboard. It does not remove the need to examine contamination exposure, oracle quality, and harness effects separately.

Models clustered within a negligible margin on Verified should be treated as effectively tied. A larger gap should be compared with known scaffolding variance and the contamination exposure associated with each model’s training cutoff before it is treated as a capability difference.

Notes

  1. Accepted at ASE 2026 Industry Showcase REAP: Automatic Curation of Coding Agent

    Provided details on Meta's REAP system, its shadow-deployment methodology, stability filtering, and the Harvest dataset of production-derived coding tasks.

  2. Can Coding Agents Reproduce Findings in Computational ...

    Supplied the AUTOMAT example in computational materials science showing that shallow test oracles miss methodological deviations even when top-level tests pass.

Deja Whitmore

← Back to Agent Bake-off