The demo is free. Production runs a meter
An AI agent passes the demo on accuracy. In production, cost per task decides whether it survives — tokens, tool calls, retries, and a loop that can spiral.
Almost every AI-agent pilot passes its demo. That is not a signal the system is ready; it is a signal the demo was cheap to run. A demo executes a handful of times, on the happy path, with someone in the room. Nobody looks at what it costs, because across a few runs the cost rounds to nothing. Production is the opposite: the same agent runs thousands of times a day, on messy inputs, with no one watching. And at that scale the number that decides whether the agent survives is not accuracy. It is one the demo never put on screen — cost per task.
What makes the meter run
Every step an agent takes is money. Each model call is tokens you pay for; each tool call is compute and, often, another metered API. While the steps are few and fixed, the bill is trivial. But an agent is not a fixed pipeline — it owns its own control flow. It decides when to re-plan, when to call a tool again, when to retry. That autonomy is what makes it useful, and it is also what turns cost from a line item into a variable.
A task that costs a cent on the happy path can, faced with an ambiguous input, re-plan several times, re-call the same tools, and in the worst case spin in a loop until something stops it. A hundred ordinary runs and one runaway can weigh the same. And unlike a process that crashes, an expensive loop throws no exception: it returns a plausible, well-formed answer and moves on. The cost grows quietly, which is the dangerous way for a cost to grow.
It isn't the model — it's the engineering around it
The reassuring part is that none of this is about model accuracy, and none of it needs a breakthrough. It is ordinary distributed-systems engineering, the same discipline you would apply to any service that fans out to other metered services. Put a budget on each task, with a hard ceiling that stops the loop instead of letting it spiral. Cache what repeats, because recomputing an identical answer is pure waste. Route easy steps to a smaller model and reserve the expensive one for the work that earns it. And make the loop observable in its spend, not just its outcome: how many calls, which tools, how many tokens, attributed to a single task. You cannot control a cost you cannot see, and most teams instrument the answer while leaving the meter dark.
The metric is cost per successful task
The honest number is not average cost; it is cost per successful task. Failed runs, dead-end retries, and loops that produce nothing still cost money, and folding them in is what separates a product with a margin from a slow leak that looks like it works. This is where accuracy and economics part ways. Accuracy gets the agent signed off in the last meeting before launch. Unit economics decide whether, six months later, it is still running — or whether it was quietly costing more per task than the work it replaced was worth.
An agent that is right most of the time and unbounded in cost is not a product. It is a demo with a bigger bill. If your agent handled ten times the volume tomorrow, would you know what it costs — or only that it still works?