Prompt Caching, Batches API, and Model Routing to Cut LLM Costs

A developer outlined three techniques to reduce LLM API costs without switching providers: prompt caching, the Batches API, and model routing. Prompt caching reuses a prefix across requests, cutting input costs by up to 90% after a break-even point. The Batches API offers a 50% discount for asynchronous processing, and model routing sends simple queries to cheaper models like Haiku while reserving expensive models like Opus for complex tasks.

Originally published on NextFuture LLM spend is an engineering variable, not a fixed bill — one that can be measured and reduced with the same rigor as query latency or memory footprint. That reframing is the starting point for three concrete levers that lower a Python-based Claude API bill without touching which model wins the benchmark chart. None of them require switching providers. Two are configuration changes to an existing request. The third is a routing decision made before the request goes out. A long system prompt, a tool list, or a RAG context block gets billed as input on every request, not written once. According to a worked example in a source writeup on prompt caching and cost control in Python, a 20K-token system prompt sent across 10,000 requests adds up to 200 million input tokens — at Opus 4.8 rates, that is $1,000 before the model produces a single output token. Verbose output compounds the problem. It costs twice: once directly, since output tokens bill at the higher rate, and again on the next turn, when that verbosity gets carried forward as input history. Fixing prompt structure and output length matters before reaching for any of the three levers below. Anthropic's prompt caching lets a request mark a prefix — system instructions, tool definitions, a large document — for reuse. The writeup's numbers: a cache read costs roughly 0.1× the base input price, while a cache write costs 1.25× with a 5-minute TTL or 2× with a 1-hour TTL. At a 5-minute TTL, two requests against the same prefix already break even 1.25× + 0.1× beats 2× paid uncached ; a 1-hour TTL needs about three requests to earn back the higher write cost. A minimal illustrative shape, not the writeup's exact code, looks like this in the Python SDK: illustrative example — adapt to your own client wrapper response = client.messages.create model="claude-opus-4-8", system= { "type": "text", "text": SYSTEM PROMPT, "cache control": {"type": "ephemeral"}, } , messages= {"role": "user", "content": user input} , print response.usage.cache read input tokens print response.usage.cache creation input tokens Two failure modes are worth checking for immediately. First, the minimum cacheable prefix is model-dependent — the writeup notes Opus 4.8 needs at least 4,096 tokens. Below that threshold, cache control silently does nothing: no error is raised, the request just shows a nonzero cache creation input tokens and no read ever follows. Second, if cache read input tokens stays at zero across requests that look identical, something in the prefix is quietly changing between calls — a datetime.now baked into the system prompt, a uuid4 near the front, json.dumps d called without sort keys=True , or a tool list assembled from an unordered set. Not every call needs a response in two seconds. Nightly report summarization, bulk document classification, backfilling embeddings metadata, and re-scoring an eval set are not latency-sensitive — and that is exactly the workload profile the Message Batches API is priced for. According to the writeup, the Batches API discounts standard token pricing by 50% in exchange for asynchronous processing: most batches finish within an hour, the hard ceiling is 24 hours, and results stay retrievable for 29 days. The two discounts stack independently — a batch of 10,000 classification calls that all share one large system prompt gets both the 50% batch discount and the cache-read discount on that shared prefix, per the writeup's own example. The cheapest tokens are the ones sent to a cheaper model. In the writeup's support-ticket example, if 80% of tickets are confidently triaged by Haiku at $1/$5 per million tokens, and only the remaining 20% escalate to Opus at $5/$25 per million tokens, the blended cost comes out to a fraction of routing everything through Opus — with no quality loss on the easy majority, because the escalation path exists for exactly the cases where the cheap model reports it isn't sure. The writeup flags one failure mode to guard against directly: a cheap model that is overconfident. Routing only works if the triage step actually escalates uncertain cases instead of guessing past them. The pressure to have this lever ready isn't abstract. A separate report tracking API pricing across vendors describes a full-blown price war erupting across every major AI provider, one where the premium tier is shrinking fast enough that GPT-4 Turbo — still in production use at some enterprises — is now, in that report's words, 'laughably overpriced' compared to what's currently available. A routing layer that can point traffic at whichever tier actually fits the task is what turns that shift into savings instead of just noise. Each lever has a matching way to lose money. Caching a prefix that changes on every request pays the write premium with zero reads — worse than not caching at all. Sending latency-sensitive traffic through the Batches API breaks the product for users waiting on a synchronous reply. And routing decisions built on a cheap model's confidence score, without an escalation path, just moves the quality problem downstream instead of removing it. Before implementing any of the three levers, pull the usage fields the API already returns on every response. They answer whether a lever will pay off before a line of routing logic gets written. CheckWhat to look atWhat it tells youCache is actually hitting cache read input tokens vs. cache creation input tokens Zero reads means the prefix isn't stable — fix that before trusting the TTL mathPrefix size vs. minimumToken count of the cached prefixBelow the model's minimum, caching silently does nothingLatency tolerance per call siteIs a synchronous response requiredAnything that can wait an hour is a Batches API candidateTriage confidence distributionCheap-model confidence scores on real trafficSets the escalation threshold before routing goes live This article was originally published on NextFuture. Follow us for more fullstack & AI engineering content.