cd /news/ai-agents/you-re-not-buying-tokens-you-re-buyi… · home › topics › ai-agents › article
[ARTICLE · art-170356] src=fromtheterminal.substack.com ↗ pub= topic=ai-agents verified=true sentiment=· neutral

You're Not Buying Tokens. You're Buying Retries.

A developer's analysis argues that per-token pricing hides the real cost of AI coding agents, showing that a task growing from 20K to 120K tokens over 40 turns processes roughly 2.8M input tokens, costing about $11.20 cold versus $1.62 at a 90% cache hit rate. The writeup also highlights llmtee, a self-hosted Go proxy with SQLite logging that tracks tokens, cost, latency and time to first token across provider APIs and enforces per-key daily and monthly budgets by returning a 429 in the provider's error format.

read6 min views2 publishedNov 16, 2026
You're Not Buying Tokens. You're Buying Retries.
Image: Fromtheterminal (auto-discovered)

Every team I know has a number for what their provider charges per million tokens, and almost none has a number for what it costs to finish a migration.

1. Nobody Sets Out to Buy Tokens. The Cheapest Turn Is the One You Skip. #

The most useful engineering document I read this week is a breakdown of what one coding task costs, and it makes a point that per-token price comparisons structurally hide: two models at the same price can cost wildly different amounts on the same job, because the one that reads the code, guesses wrong and reads it again pays for the whole conversation on every pass.

The arithmetic is worth internalizing. A task that starts at 20K tokens of context and grows to 120K over 40 turns processes roughly 2.8M input tokens — twenty times the size the conversation ever reached — because each turn resends everything before it. Finish it in 25 turns and you process 1.75M. Cache hit rate then swings the bill harder than any other dial: those 2.8M tokens cost about $11.20 cold, $1.62 at a 90% hit rate, and under a dollar at 96%. Output is the inverse problem — one output token costs about a hundred cache reads, and thinking is billed as output whether or not you see it, which is why an effort setting moves your bill more than your model choice does.

What falls out of this is not a tips list, it's a cost model with teeth. Give the agent a way to check its own work — a test, a build, a script that hits the endpoint — and you are buying turn reduction, the expensive dimension. Clearing context between unrelated tasks is free; summarizing to keep continuity costs about one request and pays for itself in roughly ten turns, so doing it just before you finish is waste. Changing tool definitions mid-session invalidates the entire cache, and the instructions file you load at startup is resent every turn forever. Enterprise averages land near $13 per developer per active day — cheap enough nobody audits it, real money at headcount.

Why it matters:

  • For ICs: Check usage at the end of your next three tasks. A low cache hit rate means something is churning the front of your context — that's a bug in how you work, not a price you pay.
  • For leaders: Stop benchmarking vendors on price per million tokens. The only comparison that means anything is cost per completed task on your codebase, over three or four real backlog items.
  • For founders: Fan-out is the expensive default — parallel agent teams in planning mode can burn several times the tokens of one session. Fan out deliberately, not by habit.
  • The durable lesson: token price is a vendor variable, token usage is an engineering variable. Only one of them is yours.

2. Everyone Is Buying Agent Compute. Almost Nobody Has a Meter on It. #

The second thing that caught my eye is how small the fix is. llmtee is a self-hosted proxy for the major provider APIs that logs every request with tokens, cost, latency and time to first token, then passes the response through untouched — the Unix tee, pointed at your model spend. One Go binary and a SQLite file, and any client that lets you set a base URL works with no code changes.

The part worth stealing is the key model. Each key gets a daily and monthly budget; when one crosses its limit the proxy returns a 429 in the provider's own error format and never forwards the request. Issue a key per agent, per developer or per CI job and you get attribution and a hard ceiling in one move — and because the proxy swaps in the real provider credential upstream, neither teammates nor agents ever hold it. Local models served through vLLM or Ollama land in the same dashboard as hosted calls, which is the first time I've seen make-versus-buy answerable from one table.

It's also honest about its edges, unlike most observability tooling: cost is only known after the response arrives, so the request that crosses the budget still completes and a key can overshoot. That's the right tradeoff — enforcing before the call means estimating, and an estimate that blocks work is worse than a small overshoot. Note what it deliberately isn't: no routing, no fallbacks, no evals, no multi-tenancy. The FinOps layer for agents is arriving as plumbing, not as a platform, which is exactly how this should go.

Why it matters:

  • For ICs: Put a proxy in front of your own keys this week. Unmetered spend is how a $200 month becomes a conversation with your manager.
  • For leaders: Per-agent keys with hard caps are the cheapest governance you'll ever deploy. Enforcement at the network boundary beats a policy document nobody reads.
  • For founders: If your product spends tokens on a customer's behalf, cost per request is a margin metric and belongs next to latency. Finding it in the invoice is finding it late.

3. The Floor Under All of This Keeps Dropping #

Then there's the supply side, where the whole calculation gets re-based under you. A write-up of end-to-end 4-bit training describes training a production model with a native 4-bit forward pass, from the first pretraining step through the final reinforcement-learning rollout. The format stores values in 3.6x fewer bits than the previous standard and runs up to 6x faster on current hardware; the swap cost about 2% in compute efficiency, bought back as an 8% speedup in pretraining and 20% in RL, with no discernible difference in reward.

The detail that reframes the economics isn't the percentages, it's one line near the end: a model this size took several weeks to pretrain a few months ago, and this one took 68 hours. That compounding is why the model you're budgeting around got 20% cheaper per token and 60% cheaper on cache reads in a single release — and it will happen again before your next planning cycle.

So be careful which optimizations you capitalize. An architecture whose only advantage is dodging today's prices has a short shelf life — the floor will come to you. What keeps paying is work on how many turns a task takes and how much context it resends, because those belong to your codebase and your habits, not to a price list.

Why it matters:

  • For ICs: Don't hand-roll clever token-shaving that makes your code worse. Shave turns, not tokens.
  • For leaders: Price any "too expensive at current rates" call against a 12-month curve. Several projects you shelved this year are affordable now and nobody has revisited them.
  • For founders: Margin that depends on prices falling isn't a strategy, but margin that ignores the curve gets undercut by someone who didn't.

The Verdict: Real or Hype? #

Cost per finished task as the real unit → Real. Turns and cache behavior dominate the bill; the per-token price you negotiated barely moves it. Metered, capped agent spend → Real but early. The tooling is a single binary and a key per agent; the practice is still almost nowhere. Low-precision training collapsing the cost floor → Real. Weeks to 68 hours is not incremental, and it reaches your invoice whether you track it or not.

── more in #ai-agents 4 stories · sorted by recency
── more on @llmtee 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/you-re-not-buying-to…] indexed:0 read:6min 2026-11-16 · —