Skip to content
Menu
Latest
Agent Bake-off
Subscribe
Search
Agent Benchmarks
Agentic IDE Reviews
Search
Find any article across the publication.
Latest articles
Agentic IDE Reviews
GitHub Copilot Workspace for Pull Request Automation
Copilot can now automate an issue through to merge approval.
Agent Benchmarks
Evaluating Agent Self-Correction and Recovery Behavior
Recovery-specific benchmarks reorder model rankings that pass/fail scores hide.
Agentic IDE Reviews
Cursor Agent Mode on Large Monorepos
Scoped rules, tight ignores, and isolation keep agents useful.
Agent Benchmarks
SWE-Bench Verified Results Interpreted for Practitioners
Understanding what SWE-bench Verified scores actually measure.
Agent Benchmarks
Reproducibility Challenges in Coding Agent Benchmarks
Four interconnected problems make coding agent benchmark scores unreliable.
Agent Benchmarks
Blind Scoring Panels for Agent Output Evaluation
Blind panels reveal how agents reach correct answers, not just whether they do.
Agent Benchmarks
Designing Task Suites That Expose Agent Weaknesses
Deliberate adversarial design catches agent failures that pass/fail benchmarks systematically miss.
Agent Benchmarks
Accuracy Metrics for Agentic Code Generation
Resolved rate conflates model capability with harness design and test quality in one number.
Agent Benchmarks
Cost per Task Comparison Across Coding Agents
Actual spend per completed task, not monthly price, reveals which coding agent costs less.
Agent Benchmarks
Speed Benchmarking Methodology for Coding Agents
Separating task completion time from model inference speed reveals what raw benchmarks hide.
Agentic IDE Reviews
VS Code GitHub Copilot Edits vs Cursor Composer Head to Head
Cursor was built around AI while Copilot was built with AI bolted on.