Cutting LLM inference cost and latency without making the feature worse

A developer outlined a practical playbook for reducing LLM inference cost and latency without degrading feature quality, centered on per-request cost and latency logging and Anthropic prompt caching. The approach routes every model call through a single wrapper that records route, model, token breakdown, time to first token, total time and computed cost, so teams can attribute spend to features and completed tasks rather than models. The writeup stresses that cache hits require byte-identical prefixes, with stable content (tools, system prompt, reference material) placed before a cache breakpoint and anything variable after it.

The first bill for an LLM feature is usually a surprise in one of two directions. Either it is tiny and nobody thinks about it again until usage grows, or it is larger than expected and someone asks for the cheapest model by Friday. Both reactions skip the useful step, which is finding out where the tokens and the seconds actually go. These are the levers I reach for, in roughly the order I pull them. The first ones are close to free. The later ones trade something away, so I only pull them with an eval running next to me. The code uses the Anthropic TypeScript SDK; the ideas apply to any provider with caching and batch pricing. I put this first even though it saves nothing by itself, because every decision after it depends on it. Each model call goes through one wrapper that records the route which product feature made the call , the model, token counts split the way the bill splits them, time to first token, total time and the computed cost. One log line per request is enough to answer most cost questions. llm/cost.ts python import Anthropic from "@anthropic-ai/sdk"; // USD per million tokens, current when I wrote this October 2026 . Keep them in config and // check them against the pricing page. Cache reads bill at about a tenth of the input rate. const PRICE: Record<string, { in: number; out: number; cacheRead: number } = { "claude-opus-5-5": { in: 4, out: 20, cacheRead: 0.2 }, "claude-sonnet-5-5": { in: 2, out: 10, cacheRead: 0.2 }, "claude-haiku-5-5": { in: 0.1, out: 0.5, cacheRead: 0.01 }, }; export function costUsd model: string, u: Anthropic.Usage, opts: { batch?: boolean } = {} { const p = PRICE model ; const usd = u.input tokens p.in + u.cache creation input tokens ?? 0 p.in 1.25 + // 5-minute cache writes u.cache read input tokens ?? 0 p.cacheRead + u.output tokens p.out / 1 000 000; return opts.batch ? usd / 2 : usd; } export function logCall route: string, res: Anthropic.Message, timing: { ttftMs: number | null; totalMs: number } { const u = res.usage; console.log JSON.stringify { at: new Date .toISOString , route, model: res.model, id: res.id, stop: res.stop reason, input: u.input tokens, cacheWrite: u.cache creation input tokens ?? 0, cacheRead: u.cache read input tokens ?? 0, output: u.output tokens, ...timing, usd: Number costUsd res.model, u .toFixed 6 , } ; } The route field is the important one. Cost per model tells you little. Cost per feature, and better, cost per completed task, tells you where to work. A cheap request that needs three retries to finish the job is not cheap. Most product prompts are mostly the same on every request: the instructions, the tool definitions, a block of reference material. Prompt caching lets you pay full price for that prefix once and a small fraction of it after that, and it also cuts time to first token because the model skips reprocessing it. The rule that makes it work is that the cache matches a prefix byte for byte. The request is read in a fixed order: tools, then the system prompt, then messages. Anything that changes between requests has to come after the last cache breakpoint, or it breaks the cache for everything behind it. Stable bytes first, a breakpoint, then everything that changes. llm/answer.ts js const res = await client.messages.create { model: "claude-sonnet-5-5", max tokens: 16000, tools: TOOLS, // same tools in the same order on every request system: { type: "text", text: INSTRUCTIONS }, { type: "text", text: PRICE BOOK, cache control: { type: "ephemeral" } }, , messages: { role: "user", content: Today is ${today}. Account: ${account.name}\n\n${question} }, , } ; The mistakes that break caching are all small. A timestamp at the top of the system prompt. A tool list filtered per user, or built from an object whose key order is not fixed. A "you are talking to {name}" line in the instructions. Each of these makes every request a cache write instead of a read. The way to catch them is the cost log: cacheRead should be large and cacheWrite small on a warm route. If both stay flat at zero, check the prefix length too, because prefixes below a model-specific minimum are silently not cached. Caching also needs traffic. The default cache entry lives for five minutes and is refreshed on every hit, so a route called every few seconds stays warm on its own. A route called twice an hour mostly misses. There is a longer one-hour option with a higher write price, which only pays off when requests sharing a prefix come in steadily with longer gaps between them. The easiest saving that does not touch quality is not using the largest model for work a smaller one does equally well. Classifying an inbound message, pulling three fields out of a form, or deciding which queue a ticket goes to rarely needs the top model. The route picks the model. No extra model call to decide. I choose the model by route, in code, not by asking another model to classify each request first. The product already knows whether a request came from the inbox sorter or from the planning assistant. Adding a model call to decide that adds latency and cost to every request, and makes the system harder to reason about. llm/routes.ts js type Effort = "low" | "medium" | "high"; export const ROUTES: Record<string, { model: string; effort: Effort } = { "inbox.classify": { model: "claude-haiku-5-5", effort: "low" }, "chat.answer": { model: "claude-sonnet-5-5", effort: "low" }, "jobs.plan": { model: "claude-opus-5-5", effort: "medium" }, }; // later: output config: { effort: ROUTES route .effort } Two cautions. Before dropping a route to a smaller model, try the same model at a lower effort setting; on current models that is often enough, and it keeps one cache instead of splitting traffic across models, since caches are per model. And every change here goes through the eval harness from the earlier post. A downgrade that drops the score is not a saving, it is a quality cut with a delayed bill. For anything a person watches, total time matters less than the time until something appears. Streaming does not make the response finish sooner, but it changes how long the wait feels. When I built a small lab to compare text-to-speech voices, the number I ended up caring about most was time to first byte, not time to complete. The same holds for text. Illustrative timings. The total barely moves; the wait does. Measure time to the first text token, not to the first event. Current Claude models think before they answer, and by default the thinking is not shown, so the stream can open well before any visible text arrives. If that gap is long on a chat route, lower the effort, use a faster model for that route, or show a progress state. llm/stream.ts js const t0 = performance.now ; let ttftMs: number | null = null; const stream = client.messages.stream { model, max tokens: 16000, system, messages, output config: { effort: "low" } } ; for await const event of stream { if event.type === "content block delta" && event.delta.type === "text delta" { ttftMs ??= performance.now - t0; send event.delta.text ; } } const final = await stream.finalMessage ; logCall "chat.answer", final, { ttftMs, totalMs: performance.now - t0 } ; A lot of LLM work in a product is not interactive: re-summarizing records overnight, tagging a backlog, running the full eval set. The batch API takes those requests, runs them asynchronously, and charges half price. Most batches finish within an hour; the limit is a day. If nobody is waiting, it should not pay for speed. Results come back in any order, so every request carries a custom id that maps back to your own record. Errors come back per request too, which means a batch job needs the same retry logic as anything else, just slower. jobs/nightly-summaries.ts js const batch = await client.messages.batches.create { requests: rows.map row = { custom id: row.id, params: { model: "claude-haiku-5-5", max tokens: 4000, system: SUMMARY PROMPT, messages: { role: "user", content: row.notes } }, } , } ; // later, once processing status is "ended": for await const r of await client.messages.batches.results batch.id { if r.result.type === "succeeded" await saveSummary r.custom id, r.result.message ; else await markForRetry r.custom id, r.result.type ; } Free wins first, trade-offs last, an eval next to every trade-off. I pull them in this order because the early ones are free and the later ones cost something. Logging and caching change nothing about the output. Effort and model choice change quality, so they go through the eval set. Streaming changes perceived latency but not the bill. Batching changes when work happens, which is only free if nobody was waiting for it. Four token counts, time to first token, total time and dollars, so you can find the expensive feature. Tools and instructions fixed, volatile data after the breakpoint, and cache reads checked in the logs. Then pick the model per route in code, and run the eval before every change. Measure time to the first text token and fix it per route. Half price for anything nobody is waiting on, keyed by custom id. None of these levers is clever on its own. What makes them work is the log line. Once every request says what it cost and where it came from, the next saving is usually obvious, and it is rarely the one someone guessed on the day the first bill arrived. Originally published on my site https://mahmadmanzoor.com/blog/cut-llm-inference-cost-latency . More posts and projects at mahmadmanzoor.com https://mahmadmanzoor.com .