# Cutting an AI agent's cost by 70%, without losing accuracy

> Source: <https://nearform.com/digital-community/cutting-an-ai-agents-cost-by-70-without-losing-accuracy>
> Published: 2026-10-22 00:00:00+00:00

### On a client engagement at Nearform, we cut the running cost of a production multi-agent system by around 70%. And according to our evals, accuracy held up through every change we made (and in some cases, it even improved).

In that system, answering a single question could cost more than $2, because each query fanned out into a chain of model calls. An orchestrator routed the question, subagents reasoned over it, and tools ran and fed their results back into the next call. Multiply that by real production traffic, and the monthly bill became something the business was asking us about.

The easy way to spend less would have been to just accept worse answers. Swap in a smaller model, strip back the context, cut a few steps, and the cost drops right along with the accuracy. But people rely on this system to answer real questions, so that trade was never on the table. Our actual challenge was trickier: cut the cost significantly *and* prove the answers were still right.

By the end, our P95 cost per trace had fallen from $2.40 to $0.50. We also didn't have to take the accuracy on faith, because every time we changed something, we checked it against a golden dataset with multiple evaluation scorers.

We got there by stacking a handful of techniques on top of each other. We measured every query so we always knew where the money was going, picked the right model for each agent, trimmed the context and cached the parts of it that didn't change, and gave the agent a shell so it could filter its own tool output. Let's walk through each of these in turn.

## Our starting point: measuring cost and accuracy

Before we changed anything, we set up measurement, since every decision after that would depend on it. For each change, we tracked two main numbers: what it cost, and whether the system still got the answer right. We kept an eye on latency throughout as well, and several of the changes in this article ended up cutting it as a side effect (smaller, faster models and less context to process both help).

For cost, we recorded what every query spent, down to the token and the dollar, so we could open up any single request and see exactly where the money went. We used Langfuse for tracing, with a custom metering layer on top for the cost accounting.

Once we could see the costs, it was pretty clear where to aim. The telemetry broke cost down per agent, and it turned out one agent alone was consuming around 65% of the tokens in the entire system. So that's where we started. Any improvement there would move the bill much further than the same effort spread across the cheaper agents, which also meant the first change we shipped was the one that mattered most.

From there, the traces kept pointing us at the next thing to optimise, usually somewhere in the trace of the most expensive query. They also made the trade-offs easy to explain to the business, not just to engineering.

Accuracy was the other number, and it's the one that decided what actually shipped. We were only able to push the cost down this far because we could measure quality at every step. Whenever we swapped the large model for a smaller one, gave the agent a shell, or trimmed a tool's output, we didn't have to guess whether accuracy held up. We just ran the golden dataset and looked at the scores. If a change saved money but hurt accuracy on important steps, it didn't ship. If you're curious how we build these evals, we've written that up in “*[From AI prototype to production: how to build evals for reliable agents](https://nearform.com/digital-community/from-ai-prototype-to-production-how-to-build-evals-for-reliable-agents/)*”.

## Picking the right model for each agent

With measurement in place, the models were the first thing we looked at. A lot of teams (ourselves included) start a new agent on the most capable model available and move on. That works, but it gets expensive. This time we started from the other end. We defined the accuracy, latency, and cost the workload actually needed, then benchmarked candidate models against those targets, so our pick was based on evidence rather than habit.

The first thing that narrowed our options was the platform. Everything here ran on Amazon Bedrock, so we could only choose from the models Bedrock offered. Without that constraint, you can reach for any model with an API, but if you're locked to one provider by contract or working inside a regulated cloud, the list gets shorter. So we worked out what our real candidates were first, and then ran the evals across them.

From there, we put two models head to head, one much larger and more expensive than the other, and measured both on the evals we'd built with the client's subject matter experts. Unsurprisingly, the smaller model started out behind on accuracy. What did surprise us was how small the gap was, and how much of it we could close without touching the model at all.

We closed that gap by working on the harness around the model. We sharpened the prompts, tightened up the tool orchestration, and moved more of the control flow out of the model and into code. With those changes in place, the smaller model matched the larger one's quality on our eval set, at a fraction of the price. It's something we keep relearning: a smaller model with a good harness, built for a specific use case, can often do the job just as well as a bigger model left to do everything on its own.

Document retrieval was a good example of how much of that harness comes down to the prompt. When we first tried the cheaper model there, the evals dropped, with retrieval accuracy (returning the right chunks) down by around 15%. The culprit turned out to be the prompt. It had been sized and structured for a much larger model, and that doesn't automatically carry over to a smaller one. Once we resized it, accuracy climbed back up to where the larger model had it, at the cheaper model's price.

Because this was a multi-agent system, we also had a knob that a single-model app doesn't have, since we could pick a different model for each agent. We saved the most capable model for work that genuinely needs reasoning, like the complex research agent, and ran the narrower jobs on smaller, cheaper models. That covered RAG search, formatting, routing, and anything else with a single well-defined task. Every subagent we moved from the top model down to a mid-size or small one took a slice out of the bill, and across a workflow with lots of calls, those slices added up fast.

How much this saved depended on how much each agent was doing. Our heavy agent, the one consuming 65% of the system's tokens, ended up needing the full toolkit: a smaller model, an aggressive cut to its context, and a sandbox it could offload data work to at runtime (the context and sandbox changes each get their own section below). A second agent with a much narrower job needed just one change. We moved it to a much smaller model and its cost dropped by around 90%. That model was also faster in tokens per second, so latency fell along with the bill, and accuracy on the eval set stayed almost unchanged.

## Trimming the context we were paying for

For the heavy agent, our next target was the context. We were paying by the token on every call, and in our pipeline the context just kept growing. Tool results piled up, and the same data got passed from step to step, so a good share of the window ended up holding fields the model never even read.

Tool design is where we saw this most clearly. A tool that reliably returns a large response floods the context on every call, often with fields that are useless for that particular turn. We had two ways to deal with that. The static approach is to shape the tool's output ahead of time, down to only the fields the agent is known to need. The dynamic approach is to give the agent a tool for filtering its own results on demand, the same way you'd reach for `jq` on the command line, so it pulls out only what that turn actually requires.

We started with the static side, shaping what went in and out of every tool. As an example, a raw OpenSearch response for a holdings query comes back with nested tax-lot metadata, repeated field names, scoring internals, and a dozen fields the agent is never going to look at. We cut that down to the fields the agent had already picked in an earlier discovery step, dropped the duplicates, and flattened what we could. For that one tool, the output shrank by more than half, and the agent could read the part that mattered instead of digging for it.

Across the pipeline, this kind of pruning cut token usage by up to 60%. It also made the agents more accurate, because a window full of irrelevant fields makes it harder for a model to pick out the right ones. Models pay less attention to whatever sits in the middle of a long context (a behaviour known as "[lost in the middle](https://arxiv.org/abs/2307.03172)"), and quality degrades as the window fills up with junk (a phenomenon called "[context rot](https://www.trychroma.com/research/context-rot)"). In practice, that's what context engineering ended up meaning for us: being picky about what went into the context, so it only carried useful information.

## Caching the parts that don't change

Even after all that trimming, a lot of what the agent sent on each turn stayed the same: the system prompt, the tool definitions, the domain guidance, and the meta-index field descriptions. We were sending all of it to the model on every call, and paying to process it every single time.

To stop paying for the same tokens over and over, we turned to [Anthropic prompt caching](https://claude.com/resources/articles/prompt-caching). Caching saves money when the same prefix gets read many times, which happens to be exactly what an agent loop does.

The tricky part was deciding where to put the cache breakpoint. A fixed breakpoint after the system prompt caches too little, because in an agent most of the bulk is the history of tool results and intermediate steps, and that history grows with every call. So instead, we moved the breakpoint along with the conversation. On each model call, we mark the last message in the conversation as the cache point. The next call extends that same prefix, so everything up to the previous mark comes back as a cheap cache read, and only the newest tokens get processed at full price.

Mastra, the TypeScript agent framework we built the system on, lets you cache the system prompt out of the box, with an option you set when you instantiate an agent. For the rolling cache point, we built a custom processor on top of that:

```
export class RollingCachePointProcessor implements Processor {
  readonly id = "rolling-cache-point";

  processLLMRequest({
    prompt,
    model,
  }: ProcessLLMRequestArgs): ProcessLLMRequestResult {
    // Target the last non-system message: that is the end of the accumulated
    // tool-result / assistant history. The system prompt already carries its own
    // cachePoint, so we never double-mark it.
    let target: (typeof prompt)[number] | undefined;
    for (let i = prompt.length - 1; i >= 0; i--) {
      if (prompt[i].role !== "system") {
        target = prompt[i];
        break;
      }
    }
    if (!target) return undefined;

    type ProviderOptions = NonNullable<
      (typeof prompt)[number]["providerOptions"]
    >;
    const providerOptions: ProviderOptions = target.providerOptions ?? {};
    providerOptions.bedrock = {
      ...providerOptions.bedrock,
      cachePoint: { type: "default" },
    };
    target.providerOptions = providerOptions;
    return { prompt };
  }
}
```

*Caching has its own costs, which we had to account for too. A cache write costs a premium over a normal read, about 1.25x the base rate for a five-minute TTL and 2x for a one-hour TTL, and it only pays for itself once a read actually lands against that entry. A five-minute TTL breaks even after a single hit, and a one-hour TTL after two. So we only turned caching on where we knew the agent would loop for at least a couple of turns, and left it off for short, one-shot exchanges.*

## Giving the agent a shell (which it already knows how to use)

That still left the tasks that needed a lot of filtering and transforming of data before an answer fell out, where the raw tool output could be enormous. Back in the context section, we mentioned the dynamic approach of giving the agent a tool to filter its own results on demand, the same way you'd reach for `jq` on the command line. Here's how we actually built that.

Models are already pretty good at using a shell, so we gave the agent one. Inside a sandboxed environment in Bedrock AgentCore Runtime, the agent can pipe a tool's raw response through `jq`, `grep`, and other standard command-line utilities and decide for itself what to keep, rather than passing every byte of a large response through its own context. The model chooses which command to run, and the sandbox executes it. This turned out to be cheaper, since we only paid tokens for the filtered result, and more reliable too, since `jq` parses and filters the same way every time (unlike a model picking through raw JSON by eye).

## The difference is engineering

The kind of query that used to cost $2.40 at P95 is still running. It does the same job, and the scores say the answers are just as good, but now it costs $0.50.

Looking back, that first bill was really just what the system cost before we'd put the engineering work into it. We scored every change on three numbers (cost, accuracy, and latency) and kept only the ones that made things cheaper without making them worse or slower. Without that measurement in place, every one of these cuts would have been a gamble with no way to tell whether it paid off. And we're not done. Cost cutting is an ongoing process, and as new models come out and the technology evolves, there are always more opportunities to improve.

If you're running an agentic system and the bill is climbing faster than the value it delivers, or you're looking for a partner who has taken one of these from prototype to sustainable production, [we'd love to hear from you](https://nearform.com/contact/).

## But wait - there's more.

Nearform publishes real-world learnings on data & AI, engineering, and digital strategy - with more merged in weekly.

### Insights

Perspectives on AI in engineering, product development, and strategy, for enterprise executives.

### Community

Deep dives and tutorials by engineers, for engineers.

## You may also like

### Web AI runtimes: five ways to run a language model in the browser

[Ryan Roemer](https://nearform.com/authors/ryan-roemer)· 8 Oct 2026 · 20 min read