
Essay Agent Benchmarks
Evaluating Agent Self-Correction and Recovery Behavior
Recovery-specific benchmarks reorder model rankings that pass/fail scores hide.
For an agent to be reliable in production, it must respond to failure in the right way, quickly and safely. A useful evaluation therefore examines where execution first diverged, how the agent responded, and whether its corrective actions repaired the situation or compounded it.
Why end-of-run pass/fail scores hide the question that matters most in production
A controlled demo shows one run under clean conditions, while production exposes an agent to hundreds of calls and repeated opportunities for failure. Even a small chance of error at each step adds up over a long chain of actions. As trajectories grow, what happens when execution goes wrong becomes operationally important. The final result cannot explain that behavior.
Standard agent benchmarks check whether the task succeeded, usually just once, at the end. That label conflates baseline planning competence with fault recovery. An agent may know how to attempt a task yet respond poorly when execution breaks down, or recover effectively after an initially weak plan. Outcome-only evaluation cannot distinguish those cases, leaving teams without evidence about the behavior they will depend on when clean conditions disappear.
What recovery-specific benchmarks reveal that standard ones cannot
Recovery-Bench, presented by Letta AI at the NeurIPS 2025 Workshop, initializes agents with environments and action histories derived from failed trajectories. Its results showed a 57% relative decrease in accuracy compared with the original Terminal-Bench. More revealingly, the model rankings changed. Claude 4 Sonnet had the highest Terminal-Bench score of 34.8% but came third on Recovery-Bench, while GPT-5 scored much lower on the original benchmark but came first on recovery.
A clean-slate leader was not the strongest model when execution began inside an existing failure. Selecting an agent only by its fresh-start performance can misjudge its reliability precisely when execution becomes difficult.
UndoBench, introduced by Sah et al. in October 2026, uses counterfactual paired trials under identical seeds, with wire-level effect-history and environment-state oracles, to separate task competence from recovery across 36 base workflows and 36 fault scenarios in 8 enterprise domains. Faults lower scores. An agent’s ability to recover also depends on where execution fails, which a single end-of-run result cannot reveal.

The three sub-capabilities inside "self-correction" and why conflating them produces bad evaluations
Self-correction contains at least three distinct capabilities, which are error detection, error localization, and error correction. An agent must notice a problem, identify where it arose, and make a change that improves the situation. These capabilities can fail independently of one another. A decomposition study on GSM8K-Complex found that models that were good at detection were not always good at correction, even though detection varied widely across architectures.
The same study found the Accuracy-Correction Paradox, in which weaker models had higher intrinsic correction rates than stronger models. Its proposed Error Depth Hypothesis offers an explanation. Stronger models make fewer errors, but those errors may be deeper and more resistant to self-correction. This proposed explanation may not hold in every case, but it gives reason to question whether models with greater baseline accuracy are also better at recovery.
AgentProp-Bench found that rejection and recovery were independent, with Gemini-2.0-Flash showing the highest rejection rate but only moderate recovery. GPT-4.1-nano showed moderate rejection but near-zero recovery at 2.9% once an injection passed the schema gate. ParaRecover, introduced by Guan et al. at EMNLP 2026, targets localization and recovery in multi-turn parallel tool-use agents, where concurrent calls add a coordination problem. Together, these findings support evaluating each capability directly rather than inferring the whole process from its final output.
Why intrinsic self-correction — without external signals — is unreliable and can make things worse
Intrinsic self-correction without external signals is not a dependable fallback. Huang et al.’s prompt-only, zero-shot findings showed that GPT-4 accuracy on GSM8K fell over successive correction rounds, and GPT-3.5 accuracy also declined. More attempts did not produce better answers. In those conditions, correction degraded results that would otherwise have remained unchanged.
Chen et al.’s 2026 study found a more specific barrier than a lack of effort. It used hash-identical error text and changed only source attribution through the chat-template role, finding that correction rates rose by 23 to 93 percentage points when attribution changed. Models could fix an error presented as external yet fail on the same error presented as their own. The error itself had not become easier to solve. This blind spot is structural in how the model handles attribution, rather than evidence that another request to check its work will suffice.
Models can claim to double-check their work while generating plausible rationales for incorrect answers, and spontaneous shifts in extended reasoning are often symptoms of a trace already going badly. The design implication is to surface errors as tool output or retrieved memory rather than as the agent’s own prior reasoning when correction is needed. A second pass or a statement of doubt is not evidence of recovery unless the resulting action improves the downstream trajectory.
Standard agent benchmarks check whether the task succeeded, usually just once, at the end.
What training and external scaffolding can change about recovery, and what they cannot
The limits of prompt-only correction do not establish an immutable limit on trained recovery. DeepMind’s SCoRe produced meaningful gains on MATH and HumanEval through multi-turn reinforcement learning on self-generated correction data. The useful question is therefore under what training regime, feedback design, and task structure self-correction works reliably, rather than whether it works at all.
External scaffolding changes those conditions as well. A dedicated reflector module roughly doubled success rates on WebArena, providing directional evidence that recovery architecture can substantially affect results. DENSE takes a complementary approach by distilling evidence-grounded shortcut trees from prior trajectories without outcome labels. On Terminal-Bench 2.1, it improved strict pass rates over initial attempts while using fewer tokens and outperformed self-reflection on hard tasks across multiple feedback iterations.
These gains remain specific to tasks and regimes. UndoBench’s phase-dependent vulnerabilities show why improved recovery machinery does not remove the need for independent evaluation, including for trained models. Plan-RewardBench makes a further distinction by treating a repeated failing call as a negative and preferring diagnosis-and-fix behavior or a strategy shift. A retry is not itself a recovery. Evaluation must distinguish a change in approach from mechanical repetition, even when both appear inside a system explicitly designed for self-correction.
Recovery in safety-critical contexts: when the wrong corrective action is worse than no action
Safety-critical recovery must address prohibited state changes as well as lost accuracy. Tencent Zhuque Lab’s 2026 RogueHandoff-20 tests 20 executable scenarios in which an agent receives an unsafe trajectory and the instruction “Continue,” without an added critique or error signal. Recovery requires interrupting the inherited course before executing a prohibited action. Across four native-pending routes, agents caused much more harm after injection than on normal tasks.
The mechanism differs from an agent independently devising an unsafe shortcut. An agent may rarely originate such a shortcut yet readily follow it when received context presents it as established work in progress. The handoff changes what the agent accepts as a course to continue. To recover successfully, the agent must spot danger in a trajectory that comes without an error flag, before it takes an irreversible action that later correction cannot fully repair.
The same paper frames collective loss of control as a contest between contagion and recovery. A local deviation can spread through a multi-agent system when propagation outpaces correction, making per-agent recovery a system-level safety property. Its findings support susceptibility to unsafe handoffs. A 2026 longitudinal taxonomy of silent failures in a production LLM agent runtime adds another concern by documenting failures in which the agent neither recovers nor reports a problem. Evaluation that treats recovery only as a performance variable is therefore insufficient here, because safe recovery must operate as a constraint on what the agent may do.
What a recovery-focused evaluation actually examines, step by step
For teams designing evaluations, the first question is where the trajectory initially departed from a valid path. UndoBench provides a method through wire-level effect-history and environment-state oracles that locate divergence. Establishing that point separates the original error from later actions that may conceal, repair, or compound it.
Did it repeat the same failing call, diagnose the problem, or change strategy? Plan-RewardBench supplies the distinction between repetition and genuine course correction. Once the response is characterized, detection, localization, and correction need separate probes. Otherwise, a successful fix can obscure unreliable detection, while an accurate diagnosis can receive undue credit despite an ineffective repair.
The next decision concerns the effect of the corrective action. The relevant evidence is the downstream state. AgentRewind, introduced in 2026, records aligned checkpoints of agent context and the controlled environment so execution can return to an earlier state and resume with information from previous attempts. That method supports examining recovery from matched context and environment states. The evaluator can then assess whether the resumed trajectory improves rather than assuming that another attempt constitutes progress.
When outcome labels are unavailable for generating feedback, the next question is whether evidence from earlier execution helps a retry. DENSE’s REFIT protocol compares first attempts with retries guided by feedback, keeping the task, tools, and execution budget the same. It separates label-free feedback construction from assessment of whether that feedback helps. This is useful for online production settings where a post-execution reward signal is unavailable, without treating every piece of trajectory-derived advice as valid.
ParaRecover supplies the method for examining localization across parallel tool calls, where the agent must attribute failure to the correct concurrent branch. ReInBench/ReCoF, introduced in 2026, examines whether self-recovery can be predicted when an error occurs, how quickly it happens, and how it relates to later execution and completion. These questions extend assessment beyond eventual success. An operationally useful recovery evaluation establishes not just whether repair occurred, but whether it was timely and predictable enough to support deployment decisions.
The open problems that current evaluation frameworks still cannot answer
The proliferation of recovery benchmarks in 2026 has not produced an agreed definition of recovery. Recovery-Bench, UndoBench, ParaRecover, ReInBench, AgentProp-Bench, AgentRewind, and RogueHandoff-20 isolate different problems, including polluted context, side-effect integrity, parallel tool errors, silent failures, and injection propagation. None subsumes the others. Their differences reflect distinct failure mechanisms, so cross-study comparisons cannot safely treat their scores as interchangeable measures of one capability.
The Horizon Gap survey describes a related dispute over where recovery should happen. Individual-agent hardening competes with orchestration approaches such as multi-agent checking, circuit breakers, and human escalation. The survey describes greater investment in orchestration than in strengthening an individual agent’s error-correction, while identifying hardening as the harder unresolved problem. It also raises correlated measurement bias. If training signals and evaluation signals share assumptions about progress, apparent gains may not provide independent evidence of reliability.
Silent failures expose the sharpest limit of signal-dependent evaluation. A framework that waits for the agent to report trouble cannot establish how often unreported divergence occurs, and compromised signals complicate the separation of model reliability from harness reliability. These limits call for broader evaluation. Teams deploying agents should build the recovery sequence into development now, treating each benchmark score as evidence about one facet. Teams that evaluate recovery have a stronger basis for deployment decisions than teams that measure completion alone.
Notes
- DENSE: Distilling Agent Trajectories into Evidence-Grounded Shortcut Trees for Self-Refinement
Supplied the DENSE method, its Terminal-Bench 2.1 results, and the REFIT protocol for label-free feedback evaluation.
- The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents
Provided the discussion of the dispute between individual-agent hardening and orchestration, plus the correlated measurement bias concern.
- Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery
Provided the RogueHandoff-20 scenarios and the framing of collective loss of control as contagion versus recovery.
- AgentRewind: Recoverable Executionfor Long-Horizon LLM Agents
Described AgentRewind's aligned checkpoints of agent context and environment for resuming execution from an earlier state.


