
Reproducibility Challenges in Coding Agent Benchmarks
Four interconnected problems make coding agent benchmark scores unreliable.
October 29, 20268 min read

Four interconnected problems make coding agent benchmark scores unreliable.
October 29, 20268 min read

Copilot can now automate an issue through to merge approval.
November 21, 20268 min read

Recovery-specific benchmarks reorder model rankings that pass/fail scores hide.
November 10, 20268 min read

Scoped rules, tight ignores, and isolation keep agents useful.
November 9, 20268 min read

Understanding what SWE-bench Verified scores actually measure.
November 5, 20267 min read

Blind panels reveal how agents reach correct answers, not just whether they do.
October 24, 20268 min read

Deliberate adversarial design catches agent failures that pass/fail benchmarks systematically miss.
October 13, 202610 min read

Resolved rate conflates model capability with harness design and test quality in one number.
October 10, 20269 min read

Actual spend per completed task, not monthly price, reveals which coding agent costs less.
October 9, 20269 min read

Separating task completion time from model inference speed reveals what raw benchmarks hide.
October 8, 202611 min read

Cursor was built around AI while Copilot was built with AI bolted on.
October 8, 202611 min read