Bigger context windows look like a free upgrade until the bill arrives. The mechanism that quietly wrecks agent margins is the KV cache, and almost nobody explains it in plain terms.
- Attention cost scales roughly quadratically with context length, so doubling your prompt can more than double compute time, not just double it
- The KV cache grows linearly with every token you keep in context, and it lives in GPU memory the whole time a request runs, which is why providers charge more once you cross fixed thresholds
- Google's Gemini 1.5 Pro pricing doubled per-token rates once a prompt passed 128,000 tokens, a real example of a provider pricing the nonlinearity directly into the API
- Anthropic's prompt caching can cut the cost of repeated context by up to 90%, but only if your agent actually reuses the same prefix instead of rebuilding it every turn
- Agents that re-send full conversation history on every step pay the quadratic cost repeatedly within a single session, which is where most founders lose their margin without noticing
Here's the assumption nearly every founder building an AI agent makes at some point: a 200,000-token context window is just a bigger box, and filling more of it costs proportionally more, in a straight line. It doesn't. Attention, the mechanism that lets a transformer relate every token to every other token, scales roughly with the square of the sequence length, not the length itself. Feed a model twice as much context and you're not paying twice the compute. You're paying closer to four times, before you even get to the part that actually kills margins on repeat-turn agents: the KV cache.
The KV cache is where the money actually leaks. Every time a transformer processes a token, it produces a key vector and a value vector for every attention layer and every attention head, and the model has to hold onto all of those to answer the next token correctly. That's the cache. It doesn't shrink. It doesn't get discarded between turns of a conversation. It sits in GPU memory for as long as the request or session is alive, and its size is a straight multiplication: tokens times layers times heads times head dimension times two, for keys and values, times the size of the number format being used. A 70-billion-parameter model with a few dozen layers running a 32,000-token context can push its KV cache into double-digit gigabytes, on top of whatever memory the model weights themselves already take. That's memory a provider has to reserve and pay for, per request, and it's exactly why the pricing you see isn't linear either.
Google didn't hide this. Gemini 1.5 Pro's published API pricing charged a higher per-token rate once a prompt crossed 128,000 tokens, roughly double the rate below that line. That's not a marketing decision. It's a provider passing the real cost of attention and cache overhead straight through to the customer, in a tier structure instead of a smooth curve. OpenAI and Anthropic don't structure their sticker pricing the same way, but the underlying compute curve is the same physics under different billing dressing, and it shows up the moment you start paying for prefill.
Prefill is the part nobody puts on a slide. When a request comes in, the model has to process the entire input context before it generates a single output token, and that step is where the quadratic attention cost actually lands. Generation, token by token after that, is comparatively cheap per step because the model is attending a new token against an already-built cache rather than rebuilding everything from scratch. So the real cost driver in a long-context agent isn't the answer. It's re-reading everything that came before the answer, every single time you call the model again.
AI Inference Costs Are Quietly Eating SaaS Gross Margins AI inference costs are the hidden line item turning great LLM features into shrinking SaaS gross margins, because unlike traditional software, every query costs money and scales linearly with usage instead of flattening out. This piece breaks down why the unit economics break and the concrete levers, caching, model routing, quantization, and... - how to reduce AI inference costs - SaaS gross margin decline from AI
That's the trap for agents specifically. A chatbot that answers one question and closes the session pays the prefill cost once. An agent that loops, reasoning through a task across ten or twenty tool calls, and re-sends the growing transcript as context on every single step, pays a version of that prefill cost again and again, with the context getting longer each time. Turn one might cost you the equivalent of a 500-token prompt. By turn fifteen, if you're appending every tool output and every intermediate thought back into context, you could be re-processing 15,000 tokens of history just to ask the model what to do next. The growth isn't linear across the session. It compounds, because each new turn pays for everything the turns before it already paid for, plus its own increment.
Character.AI ran into this exact wall years before most startups were building agents at all, and they wrote about how they fixed it. In a 2023 engineering blog post, the company described cutting inference costs by roughly 33x through a stack of changes that included multi-query attention, which shrinks the KV cache dramatically by sharing key and value projections across attention heads instead of giving every head its own copy, plus int8 quantization and a hybrid attention horizon that limited how far back the model actually needed to look. They weren't optimizing for elegance. They were serving millions of daily conversations where every extra gigabyte of cache per session was real infrastructure spend, and the math didn't work until they attacked the cache directly rather than just buying bigger GPUs.
Where the AI agent context window cost actually breaks your margin #
Run the numbers on a real agent pattern. Say your agent charges a customer $0.02 per completed task, and a task takes eight model calls, each one re-sending the full running context because that's the simplest way to build an agent loop. If the first call is 1,000 tokens and each subsequent call adds another 1,500 tokens of tool output and reasoning, your eighth call is processing over 11,000 tokens of prefill, most of which the model has already seen and paid for once. Sum the prefill across all eight calls and you're not paying for 11,000 tokens of work. You're paying for something closer to 45,000 tokens of cumulative reprocessing, on a task you priced as if it cost 1,000. That gap is where the unit economics collapse, and it happens quietly because each individual call still looks cheap on its own line item.
Frankly, most teams don't notice until the bill lands at scale, because a single test run in a demo never surfaces the compounding. It only shows up once you're running thousands of sessions a day and someone in finance asks why inference spend tripled without a corresponding jump in usage.
How to reduce token costs on large context without gutting the product #
The fix isn't shrinking the context window your customers see. It's stopping your agent from paying full prefill price for context it has already processed before. Prompt caching is the most direct lever available right now. Anthropic's caching lets you mark a stable prefix, a system prompt, a knowledge base excerpt, a tool schema, so that repeated calls reuse the cached version instead of reprocessing it, at a claimed discount of up to 90% on those cached tokens. OpenAI offers a comparable caching discount on repeated prompt prefixes in its API. The catch is that caching only helps if your context is actually structured with a stable, reused prefix and a small variable tail. An agent that reshuffles or rewrites its history on every turn gets none of the benefit, because the cache only matches on an identical prefix.
The second lever is retrieval instead of accumulation. Don't append the full transcript of everything the agent has done. Summarize completed steps, keep only the tool outputs that are still relevant to the next decision, and pull older context back in through retrieval only when it's actually needed. This is more engineering work than just letting the context grow, and it's also the only approach that keeps cost roughly flat as a session gets longer instead of climbing with it.
The third lever is matching model tier to task. Not every step in an agent loop needs your best model reasoning over your full context. Routing simple tool-selection or formatting steps to a smaller, cheaper model, and reserving the large-context, expensive model for the step that genuinely needs the whole history, is a routing decision, not a model quality decision, and it's one most agent frameworks still don't do by default.
How Does Prompt Caching Work for LLMs, and Why It Cuts Bills in Half How does prompt caching work for LLMs? It's a token-storage system with its own write and read pricing, its own expiration clock, and a hidden minimum size that fails silently if you miss it. Get the prompt order wrong and you pay full price every time, even with caching turned on. - how prompt caching works for language models - why prompt caching reduces LLM API costs significantly
None of this means long context windows are a bad idea. A 200,000 or 1,000,000-token window genuinely lets you build agents that couldn't exist otherwise, ones that hold an entire codebase or a full case file in view at once. The mistake is pricing your product as if the window is free just because the provider advertises it as available. It's available. It's also the single line item most likely to eat your margin if your agent architecture pays full price for it on every turn.
Also read: How Does AI Model Routing Work, and Why It's Halving LLM Bills • How Reg CF Equity Crowdfunding Actually Works for Startups Under $5 Million • How the R&D Tax Credit Offsets Payroll Taxes for Startups With No Revenue
This article is posted in AI News, check it out for more related stories.
Join the discussion #
Open in the community → Almost there. Sign in and your reply posts straight away.