Cover illustration for “Cost per Task Comparison Across Coding Agents”

Essay Agent Benchmarks

Cost per Task Comparison Across Coding Agents

Actual spend per completed task, not monthly price, reveals which coding agent costs less.

Cost per task, not monthly subscription price, is the number that actually separates one coding agent from another. The rest of this piece explains why that number moved to the center of the conversation in 2026, what sets it, and how the major tools stack up once you measure it honestly.

Why monthly subscription price stopped being a useful way to compare coding agents

Every major coding agent advertises an entry price in roughly the same band, so the tools can look interchangeable on a pricing page, but they are not. Production developers routinely spend far more than that sticker number once real work starts. The 2026 AI Coding Agent Cost Ladder documents heavy Claude Code API users reporting monthly spend many times above the plan price, a gap wide enough to turn a modest line item into a substantially larger one even when the developer changes nothing about how they work. Cursor Pro customers learned a harder version of this lesson in June 2025, when a billing model change led one team to burn through a $7,000 annual subscription in a single day of normal use. None of the mainstream tools surface cost per completed task by default, so two agents can carry near-identical monthly prices while actual spend diverges by an order of magnitude once real usage starts. If the subscription price does not tell a buyer what a tool costs to use, the question becomes what does, and what sets that number.

What cost per task actually measures and why it is the right unit

Cost per task is the total billed spend you need to produce one accepted code change. That includes retries, the token overhead of long context windows, failed attempts that went nowhere, and any human time spent cleaning up afterward. It does not mean the price of the tokens in a single successful call. OpenHands draws this line explicitly: token price is not task cost, because retries, long contexts, failed attempts, and human follow-up all add to the bill, so the number worth tracking is spend per accepted change. The gap between those two figures is not small. SWE-Effi research found an "expensive failures" pattern in which unresolved attempts consume on average four times more resources than successful ones. A cheaper model that fails more often can end up costing more per accepted outcome than a pricier model that gets the task right on the first try. SWE-Effi also found that agentic coding tasks consume roughly a thousand times the tokens of single-turn chat or reasoning workloads, and that token usage for the same task varies widely across repeated runs. A one-shot number pulled from a vendor demo will understate the real range a team should expect. The practical move is to collect billed usage across a sample of representative work, record whether each task was completed and to what quality, and compare total cost per successful outcome.

The three drivers that set cost per task: model tier, harness overhead, and billing structure

Diagram: Same Task, Five-Times Cost Spread by Model Tier. Visualizes: Show a single horizontal bar chart (or ranked magnitude visual) comparing the illustrative token-basket cost for the same standard task across three model tiers: Claude Haiku 4.5…

Nearly all the variance in per-task cost documented across 2026 traces back to three factors: which model tier handles the task, how much token overhead the surrounding harness adds on top of the model's own work, and which billing structure the vendor uses to charge for it.

Model tier sets the floor. An illustrative token basket built around a standard task (200,000 input tokens, 30,000 output tokens) shows Claude Haiku 4.5 running near $0.35, Claude Sonnet 5 near $1.05, and Claude Opus 5 and Opus 5.5 running higher still, a five-times spread on the same modeled job set purely by which model answers it. That basket is illustrative rather than a measured benchmark: real token counts shift by model and by harness, so the actual spread in production work tends to run wider than this basket shows, not narrower.

Harness overhead is the second lever, and it operates independently of model choice. SWE-Effi finds that input tokens dominate total cost in agentic workloads, and accuracy peaks at intermediate context budgets rather than the largest ones available, so a harness that stuffs more context into every call does not reliably buy better outcomes and often just raises the bill. SWE-ContextBench research points to the same mechanism from a different angle: accurate summarization and retrieval of past experience can meaningfully improve resolution accuracy while cutting both runtime and token cost, but models do not always use that capability well and can be thrown off by incorrect assumptions carried over from prior steps. How well a harness manages context is turning into a cost variable in its own right, separate from which underlying model it calls. Techniques like context compaction and prompt caching can claw back a real share of wasted tokens before a team needs to justify an upgrade to a pricier plan.

Billing structure is the third driver, and it shapes risk as much as it shapes price. Vendors have settled on three distinct mechanisms: a subscription with a usage cap, a credit pool priced at API token rates, and compute-time units billed by the job. Each carries its own failure mode. A subscription cap shifts cost onto developer time once the cap is hit and work has to wait or move to a slower path. A credit pool exposes a team to surprise overages, since one large operation, a big code review for instance, can drain the pool without warning. Compute-time units, the model Devin uses under the name Agent Compute Units, are the only structure among the three that lets a team know roughly what a task will cost before the task starts.

Cost per task is the total billed spend you need to produce one accepted code change.

Why benchmark scores are an unreliable proxy for cost-effectiveness

A higher benchmark score does not reliably predict a lower cost per accepted task, and the benchmarks themselves carry structural problems that break the link between score and real-world cost-efficiency. Research that tracks output tokens, wall-clock duration, and dollar cost per trial alongside pass rate finds that all three cost-shaped measures swing by an order of magnitude across agents, yet none correlates strongly with how many tasks actually get solved. Emitting more tokens, running longer, or spending more money does not consistently translate into more completed work. Work examining the limitations of a widely used coding benchmark adds three further problems on top of that. Issue descriptions in the benchmark are often more detailed than what a developer would write in practice, which inflates resolution rates relative to real conditions. The benchmark leans heavily on a single programming language, which limits how far its results generalize to other codebases. And scaffold and model effects are confounded in the published results, making it hard to tell whether a strong score reflects a better model or just a better-tuned harness wrapped around an ordinary one. Benchmark scores still have a place for rough orientation, telling a buyer roughly where a tool sits relative to others, but they should not be read as a cost-efficiency ranking. The number that reflects both accuracy and spend at the same time is cost per accepted change measured against real workloads, not a leaderboard position.

How the Three Drivers Play Out Across Phantom Farm and the Major Coding Agents

Running the three drivers, model tier, harness overhead, and billing structure, against the tools teams are actually weighing produces a cost-effectiveness ranking that looks quite different from a ranking built on subscription price alone.

Phantom Farm's position starts from how it treats the three drivers as a single design question rather than three separate pricing decisions bolted together. Where several of the tools below tie a buyer to one model family regardless of what a given task actually needs, Phantom Farm's approach to model tier is to match the tier to the task rather than defaulting every job to the most expensive model available, which matters directly for cost per accepted change: a routine fix does not need the same tier as a genuinely hard multi-file refactor, and paying Opus-level rates for Haiku-level work is exactly the kind of waste the token basket above makes visible. On harness overhead, the same logic that other research on agent cost and context documents, that accuracy peaks at intermediate context budgets and that smarter retrieval of past work beats simply loading more tokens into every call, is what Phantom Farm's harness is built to manage. And on billing structure, a model that ties spend to the forecastable cost of a task rather than to a usage cap that quietly shifts cost onto developer time, or a credit pool that can be drained by one large operation, gives a team the one thing a monthly subscription price can never promise: a number it can predict before the work starts, not after the invoice arrives.

Claude Code runs across Anthropic's own model families, Haiku, Sonnet, Opus, and Fable, giving it real range in model tier within a single tool. That range shows up in the benchmarks: the October 7, 2026 Terminal-Bench 4.0 leaderboard places Claude Code running Opus 5.5 and Claude Code running Sonnet 5.5 in the top two positions overall. The Max plan bills at a per-developer monthly subscription cap plus API usage on top, and the Cost Ladder puts Claude Code Max in the low-to-mid single digits per medium task while leading the field on accuracy among the compared agents. Heavy API users, though, can run up costs in the hundreds to low thousands of dollars a month, the same multiple documented industry-wide, and the Artificial Analysis Coding Agent Index v1.5, published September 22, 2026, places Claude Code paired with Fable 5.1 in the top capability tier but considerably more expensive per task than mid-tier alternatives. Claude Code only points at Anthropic models, so every pricing decision Anthropic makes flows straight through to a team's per-task cost with no alternative model to route around it.

Cursor, running Composer 2.5, bills through a credit pool priced at API token rates. The Cost Ladder places Cursor Pro in a low-to-moderate range per medium task, which looks competitive on paper, but the pool structure carries the overage risk described earlier: a single large operation can draw down the pool faster than a flat-rate plan would ever allow. Cursor offers some model flexibility, routing through Claude and GPT-5.5 among other options, which gives a team partial control over which tier handles a given job. That flexibility is bounded by the fact that billing still passes through Cursor's own layer on top of the underlying model cost.

Codex CLI, built around OpenAI's GPT-6.1 Sol as its current default model, offers the clearest documented case of equivalent benchmark performance at sharply different price points. Terminal-Bench 4.0 results put Codex running GPT-6.1 Sol at 58.2%, matching GPT-6 Astra's score on the same benchmark at roughly a fifth of Astra's cost per task. Billing moved to a token-based model as of April 23, 2026, bundled into ChatGPT plans or billed directly at API rates, and the Cost Ladder places it in a low-to-moderate range per medium task. The Codex CLI client itself is Apache-2.0 open source, though model lock-in still runs to OpenAI's GPT-6.x family, currently defaulting to GPT-6.1 Sol.

GitHub Copilot carries the cheapest headline number of any tool in this comparison, by the Cost Ladder's pricing. That number stopped reflecting real cost once GitHub switched Copilot to an AI Credits system on June 1, 2026. Copilot's accuracy is the lowest in this comparison, and lower accuracy does not vanish as a cost: it moves onto developer review time that never appears on a billing statement. Copilot's position in a straight price comparison is misleading for exactly this reason: the subscription price and the real cost of using it are two different numbers, and the gap between them is carried by the people reviewing the agent's work.

That structure, among the three billing mechanisms in use across the industry, is the only one that lets a team know the rough cost of a task before the task runs, trading the flat predictability of a subscription for cost that tracks the actual compute a job consumes.

Lined up against each other this way, the ranking that emerges has little to do with which tool advertises the lowest monthly price. It has everything to do with which model tier handles the work, how much a harness wastes on overhead before the model ever touches the real problem, and which of the three billing structures determines how a team finds out what it owes.

Priya Nandakumar

← Back to Agent Bake-off