Why your agent gets slower and pricier with every tool call

A developer built a plain-NumPy transformer to measure where agent latency and cost come from, finding that without a KV cache per-token time climbs from about 12 ms at step 16 to 550 ms at step 1023, versus 3–16 ms with caching — 232 seconds versus 12.5 seconds for 1,024 tokens, roughly 18x. The engineer notes the toy CPU benchmark's constants don't transfer to GPU-served LLMs but that the growth shape does, and ties it to prompt caching: providers that reuse a cached prefix skip reprocessing the whole conversation on each agent turn, cutting both latency and billed input tokens.

Okay so here's something that bugged me for a while. You build an agent. Step 1 feels snappy. By step 15, after a few tool calls and a pile of JSON results, it's noticeably slower, and the bill is climbing in a way that doesn't feel linear. I knew the vague answer "the context is longer" . I didn't really get it. So i stopped reading and wrote the thing myself, in plain NumPy, to see where the time actually goes. Short version: the culprit is a boring little thing called the KV cache , and once you see it, a lot of agent design advice suddenly makes sense. A transformer generates text one token at a time. For every new token, attention needs the Key and Value vectors of every token before it . That's how the new token "looks back". Here's the catch. Those old Keys and Values don't change. Token 12's Key is the same whether you're generating token 13 or token 913. So a naive loop does something silly: at every step it recomputes K and V for the whole sequence, from scratch, then throws them away. Then does it again next step. The KV cache is just... not doing that. You compute K and V for a token once, keep them in memory, and append. Here's the toy model. 4 layers, 8 heads, 512 wide, random weights. It's not smart. It doesn't need to be, we're timing the machinery. No cache, the full thing re-runs every step: python def forward full X : h = X t = h.shape 0 mask = np.triu np.full t, t , -1e9, np.float32 , 1 for w in W: n = rms h q, k, v = split n @ w m for m in "qkv" s = q @ k.transpose 0, 2, 1 / np.sqrt HD + mask h = h + merge softmax s @ v @ w "o" return h -1: With a cache, only the newest token goes through the projections: python def forward cached x, cache : h = x for w, c in zip W, cache : n = rms h c "K" = np.concatenate c "K" , split n @ w "k" , 1 c "V" = np.concatenate c "V" , split n @ w "v" , 1 q = split n @ w "q" s = q @ c "K" .transpose 0, 2, 1 / np.sqrt HD h = h + merge softmax s @ c "V" @ w "o" return h Same math. Before timing anything I checked both versions produce the same output over 40 steps max difference around 1e-6 . Honestly this was the step i almost skipped, and my first version of the benchmark failed it, the "no cache" path had a bug and I was about to publish a fake speedup. Check your baselines. I generated 1,024 tokens each way on my laptop CPU. Without the cache, the time per token keeps climbing: about 12 ms at step 16, 174 ms at step 512, 550 ms at step 1023. With the cache it stays low, 3 ms early on and about 16 ms at the end. Total time for all 1,024 tokens: 232 seconds vs 12.5 seconds . Roughly 18x. Two things worth being honest about. First, this is a toy on a CPU in NumPy. A real LLM on a GPU has different constants, and nobody should quote my 18x as if it applies to GPT-anything. The shape is what matters: one curve grows with length, the other mostly doesn't. Second, the cached line isn't flat. It still creeps up 3 ms to 16 ms . That's because the new token still has to attend over everything in the cache. The cache removes the recomputation , not the reading . This is the part I actually care about. An agent loop looks like this: model call, tool call, append the result, model call again. Every iteration sends the whole conversation back, system prompt, tool schemas, history, all of it. If the provider has to process that whole prefix from scratch each time, you pay for it each time, in latency and in input tokens. That's the "slower and pricier with every tool call" feeling. Prompt caching is the same idea one level up. Providers keep the KV cache for a prefix they've seen, so if your next request starts with the same prefix, they skip that work, and usually bill it at a discount. Which has some very practical consequences: Not glamorous. But it's the difference between an agent that costs cents and one that costs dollars. Fine, so just cache everything? The cache lives in memory, and it grows with every token. The formula is simple: bytes per token = 2 K and V x layers x kv heads x head dim x bytes per number For a Llama-2 7B style model that's 512 KB per token . At 128k context, one single request needs about 64 GB just for its cache. That's before the model weights. That's why the next thing people ask is "how do we make this smaller?" and the answer is a neat trick with attention heads. I'll cover it in the next post. If you want to poke at it yourself, the whole benchmark is about 80 lines. Change the layer count or the length and watch the curves move. Tell me if you get different shapes, i'm curious.