Who spent all the tokens? Real-time, run-scoped cost control for AI agents TokenOps, an open-source system from The Agent Plane, provides real-time, run-scoped cost control for AI agents, addressing the problem of runaway token spend across multi-call runs. Unlike traditional budget caps and gateways that operate per request or after the fact, TokenOps tracks costs per run, enabling live stopping or redirecting of expensive agent executions. The system attributes every call to a run and supports budgets at run, agent type, and user segment levels, with in-path enforcement and steering capabilities. An agent run can make many model calls, and each one costs tokens. And while one call is cheap, a run can require hundreds. It’s easy to lose track of the sum across the whole run. If an agent gets a weak result from a tool, it might retry, then retry again. A few hundred model calls later, that run may have spent more tokens than the task was worth. No single model call looks wrong, and when the monthly bill arrives, there’s no way to say which run caused it to balloon. We’re getting past the era of tokenmaxxing, where the common fix for a weak run was to give it more: more reasoning, more retries, more sub-agents, more context. But this additive reflex has become the default. The goal should be to get a high-quality result. But you also want to keep your agent runs efficient. To do that, you need to have two systemic capabilities: seeing where a run’s money went and stopping or redirecting a run while it’s still live. This post is about creating that real-time, per-run cost control, using a system we built called TokenOps https://github.com/theagentplane/tokenops as the example. Why a budget cap isn’t enough The usual control is a monthly budget cap on an API key or a team. But a budget cap can’t stop a runaway, for a simple reason: it acts on a total that is only known after the fact. The cap trips after the month’s spend is tallied, which happens after the run that overspent has already finished. In the end, a budget cap is a fail-safe. It’s meant to prevent your cost from going over a limit, but it’s by no means an optimization strategy. And when you want to extract more value out of the tokens you’re paying for, you need to optimize the run as a whole rather than just bound it. Four facts anchor our argument: Tokens are the unit of cost. Track both cost per token and value per token. Cost is created at each call. If you can’t see a call, you can’t control it. Without attribution, there’s no control. Every call should be tagged to its run and attributed to some component of your agent loop. LLM calls don’t happen on their own; they are a byproduct of your agent setup, your RAG quality, your tool structures, etc. The bill lags. To stop overspending, you must act during the run, not after it. A gateway observes an isolated request. A run spans many. Most cost guard tools are gateways, which see only one unit: a completion request, keyed by an API key. A gateway caps, routes, and meters per request. That’s the right unit when a request is the whole job. But an agent run isn’t one request. It’s a graph: a loop that retries, context that grows, sub-agents, a hand-off to a second service over the network. A gateway has no model of that graph. It sees isolated requests hitting an endpoint and doesn’t know that they belong to the same run, especially when the run spans several keys, providers, processes, or queues. That’s the core limit, and it isn’t a missing feature: the gateway’s unit of accounting is the key, and the money is spent by the run. The fix is to carry a run id with the execution graph, so every call, wherever it runs, is attributed to the same run and counted against one budget. Here’s how some common tools scope their budgets today: LiteLLM | Portkey | Cloudflare AI Gateway | Langfuse | TokenOps | | |---|---|---|---|---|---| Scope | Request / project / user | Request / key / workspace | Request / user / team / app | Trace / session / user | Run / agent type / user segment | Budget | Project / user | Key / workspace / provider | User / team / app / model / provider | — | Run / agent type / user segment | Stop | Yes | Yes | Yes | No | Yes | Steer | Routing / fallback | Routing / fallback | Routing / fallback | No | Full | Enforce | In-path | In-path | In-path | Out-of-band | In-path | LiteLLM is the closest to the landscape we describe above. Its per-team budgets can cap a multi-step loop, not just one call as long as the calls use the same key for the team. Another thing to note is that much of the savings comes from routing, not pure rate-limiting. FrugalGPT 2 sends easy work to cheaper models by trying them first, RouteLLM does the same by predicting difficulty up front, and Cost-of-Pass finds that inference-time tricks often don’t pay for themselves. Routing lowers the expected bill, but it doesn’t bound the worst case—the runaway—which is what cost control is for. The two are complementary. Placed on two axes, enforcement in-path vs. observability out-of-band and per-request vs. per-run, most existing enforcement tools operate at the request level. TokenOps occupies the in-path, run-scoped quadrant: it evaluates cumulative cost across an agent run and can enforce policy before the next action executes. The shape isn’t new: CaMeL uses a deterministic layer outside the model to enforce security constraints. TokenOps applies the same architectural idea to cost. Per request | Per run | | |---|---|---| Enforcement in-path | LiteLLM / Portkey / Cloudflare AI Gateway | TokenOps | Observability out-of-band | Langfuse | — | Deciding out of band, enforcing in the path The design separates two things: deciding what to do and doing it. The deciding is reading the run’s spend, evaluating policies, choosing an action. It’s prepared out of band, off the request’s critical path. The doing is a thin wrapper around the model client, the tool methods, the RAG pipelines, and other nodes of your agent graph. Before each call, the wrapper reads the run’s current spend and applies the decision the policies have generated: allow the call, shrink it, reroute it, or refuse it. This is where latency has to be taken into account. The wrapper does read the run’s spend before every call. In a single process, that read is in memory and effectively free. Across processes, the count lives in a shared store, so a strict, fail-closed check costs one read of that store per call—a network round trip you can measure. The default avoids that on the hot path: it reads a local copy of the count and reconciles the true spend to the shared store just after the call, which is cheap but eventually consistent. Either way, the policy evaluation is precomputed, so the in-path work is a counter read and a comparison, not a policy run. The control plane does three things with the events the agents emit: instrument record each call , account add it to the run’s ledger , enforce apply a policy . Enforcement has two moves: STEER keeps the run going by changing the next call: a cheaper model, a shorter prompt, fix a broken loop. HALT is a circuit breaker. Once it trips for a run, every later call on that run is refused until an operator resumes it. Steering runs first. A halt is the last resort because halting a legitimate run causes the outage you were trying to avoid. Wrap the provider client once. The agent code does not change. from tokenops.control.integration import wrap complete from tokenops.control.config import build governor governor = build governor store.governance config for agent , price, ApplyControls complete = wrap complete governor, controls, attr, provider=provider, model=model, dispatch=provider.complete, service="planner", each call: read the run's spend, apply the decision, dispatch, then record the true cost def governed p, m, messages : controls.begin call request = CallRequest attr=attr, provider=provider, model=m, estimated input tokens=estimate messages , max output tokens=controls.call.max output tokens governor.pre call request read spend, apply STEER/HALT decision use model = controls.call.model override or m messages = consume carry controls, messages apply queued INJECT text governor.ledger.admit seg try: response = traced p, use model, messages, max output tokens=controls.call.max output tokens return response finally: governor.ledger.complete seg record true cost back to the shared ledger Because every call goes through the tokenops wrapper, every call is tagged with its run, its agent, and its step and then written to the shared ledger. That’s what lets you read a run’s spend back as a tree, which agent and which call spent what, instead of a single number on a bill. We call that token lineage. The first bug: Two agents, one run, two budgets The test bench we used runs two agents in separate processes: a research agent that loops on a search tool and then hands its findings to a summarizer agent over HTTP. They shared one run id, and the run had a budget. It passed every test, but there was a bug. Each process kept its own spend counter in memory. Both started at zero, and both saw the full run budget, so a $2 run cap was really $2 for the research agent and $2 more for the summarizer. Together they could spend twice the cap while every local check passed. It’s a distributed-systems bug: a budget with no single source of truth. The fix was to move the counter out of each process into one shared ledger that every agent reads and writes, so the budget is measured against the run, not against each agent’s private view. After the fix, the research agent’s spend lands in the shared ledger as soon as it’s made, so when the summarizer starts, the remaining budget already reflects what research has spent. If the worst case of the summarizer’s next call wouldn’t fit what’s left, the “pre call worst case” policy refuses that call and halts the run before the summarizer spends anything. The general lesson: in a multi-agent run, a budget that each agent counts on its own is not a run budget. A run budget must be counted in one shared place so that you can attribute it to a broader use-case that drives value. What a run’s cost is made of Before any policy, count what a run costs. The bill is the sum of a completion call’s input and output tokens at their prices, multiplied by the number of calls your agent run makes: cost of a run = sum over calls of input tokens x input price + output tokens x output price calls per run = loop depth x fan-out breadth sequential steps parallel sub-agents So two things set the bill: the tokens in each call and the number of calls. The number of calls has two sources: a loop going around depth and sub-agents fanning out breadth . And the ledger needs to see and attribute every call to its run. That gives a short, closed list of things to guard: the tokens per call, the loop depth, the fan-out, and a ceiling on the total. “cost budget” is the ceiling; the other nine policies each guard one part before the run cost explodes. Policy | Guards | What goes wrong | What it does | |---|---|---|---| cost budget | the total | run spend reaches the cap | halt the run | pre call worst case | per-call | the next call’s worst case would not fit the budget | cap its output or halt | cost guard | per-call | spend crosses about 80% of the budget | steer cheaper downgrade, trim , once per run | step cap | loop depth | calls pass a per-run limit a loop | halt the run | progress guard | loop depth | the same action and result keep repeating | inject a correction, then halt | output runaway | per-call | a degenerate, repeating stream | cancel, then a bounded retry | tool fix | loop depth | a malformed or unknown tool call | inject a synthetic error, halt on repeat | context compaction | per-call | the prompt approaches the context ceiling | dedup and pin the prompt | tool output cap | per-call | a tool returns an oversized payload | store it, pass back a handle | concurrency cap | fan-out | too many calls in flight at once | queue or reject | The bug earlier was a failure of the substrate under this list: if the ledger can’t count the run, the ceiling is meaningless. Attribution comes first; every policy depends on it. Enforcement is deterministic. Each policy is a detector and an action: the detector reads the ledger and raises a signal, and the action responds. No model runs in the enforcement path as of today because you don’t want the thing being governed deciding whether to stop itself. CaMeL 11 uses the same approach against prompt injection: a deterministic layer outside the model. Policies run before a call pre call , after each call observe , or on a timer tick and choose one of eight actions: allow, mutate, inject, reject, queue, retry, cancel, halt. Two of the per-call guards bound reasoning tokens, which is not a hack. Work on test-time compute Snell et al., 2024 shows reasoning is a budget you can allocate, and s1’s budget forcing is a deterministic way to cap a model’s thinking. Overthinking is measured: on a trivial question, reasoning models have used about 20 times more tokens than a plain model with no gain in accuracy Chen et al., ICML 2025 . The list was derived from cost, but it also matches the failure studies. OWASP’s Unbounded Consumption, Microsoft’s agentic-failure taxonomy v2.0 , and Berkeley’s MAST name the same modes: loops, no-progress repetition, degenerate output, oversized context and payloads, bad tool calls, and unbounded fan-out. Reaching the same list from two directions is good evidence that it isn’t missing an obvious mode. What it doesn’t cover is correctness: MAST found more than half of multi-agent failures are specification and verification problems, and a cost cap can’t tell a right answer from a wrong one. That needs a judge model, which we keep out of the enforcement path. These policies halt on spend, loops, and stalls, not on being wrong. Running it against a real runaway To see it work, you need a real runaway. The test bench ships a leaky scenario to validate this: real web results from the search tool, but intentionally garbled, with text cut off and prices masked. The research agent can’t finish them, so it loops on the search tool. A plain step counter wouldn’t catch this well: the task might loop three times or 30, and one step can spend very different amounts depending on its context and tool output. Counting steps does not count dollars, and a reasoning-heavy step can burn thousands of tokens in a single step. Under the policies, the run climbs toward the cap and the guards fire in order. One governed run’s event log: php governed run, event log run