{"slug": "prompt-caching-batches-api-and-model-routing-to-cut-llm-costs", "title": "Prompt Caching, Batches API, and Model Routing to Cut LLM Costs", "summary": "A developer outlined three techniques to reduce LLM API costs without switching providers: prompt caching, the Batches API, and model routing. Prompt caching reuses a prefix across requests, cutting input costs by up to 90% after a break-even point. The Batches API offers a 50% discount for asynchronous processing, and model routing sends simple queries to cheaper models like Haiku while reserving expensive models like Opus for complex tasks.", "body_md": "Originally published on[NextFuture]\n\nLLM spend is an engineering variable, not a fixed bill — one that can be measured and reduced with the same rigor as query latency or memory footprint. That reframing is the starting point for three concrete levers that lower a Python-based Claude API bill without touching which model wins the benchmark chart.\n\nNone of them require switching providers. Two are configuration changes to an existing request. The third is a routing decision made before the request goes out.\n\nA long system prompt, a tool list, or a RAG context block gets billed as input on every request, not written once. According to a worked example in a source writeup on prompt caching and cost control in Python, a 20K-token system prompt sent across 10,000 requests adds up to 200 million input tokens — at Opus 4.8 rates, that is **$1,000** before the model produces a single output token.\n\nVerbose output compounds the problem. It costs twice: once directly, since output tokens bill at the higher rate, and again on the next turn, when that verbosity gets carried forward as input history. Fixing prompt structure and output length matters before reaching for any of the three levers below.\n\nAnthropic's prompt caching lets a request mark a prefix — system instructions, tool definitions, a large document — for reuse. The writeup's numbers: a cache read costs roughly **0.1×** the base input price, while a cache write costs 1.25× with a 5-minute TTL or 2× with a 1-hour TTL. At a 5-minute TTL, two requests against the same prefix already break even (1.25× + 0.1× beats 2× paid uncached); a 1-hour TTL needs about three requests to earn back the higher write cost.\n\nA minimal illustrative shape, not the writeup's exact code, looks like this in the Python SDK:\n\n```\n# illustrative example — adapt to your own client wrapper\nresponse = client.messages.create(\n    model=\"claude-opus-4-8\",\n    system=[\n        {\n            \"type\": \"text\",\n            \"text\": SYSTEM_PROMPT,\n            \"cache_control\": {\"type\": \"ephemeral\"},\n        }\n    ],\n    messages=[{\"role\": \"user\", \"content\": user_input}],\n)\nprint(response.usage.cache_read_input_tokens)\nprint(response.usage.cache_creation_input_tokens)\n```\n\nTwo failure modes are worth checking for immediately. First, the minimum cacheable prefix is model-dependent — the writeup notes Opus 4.8 needs at least 4,096 tokens. Below that threshold, `cache_control`\n\nsilently does nothing: no error is raised, the request just shows a nonzero `cache_creation_input_tokens`\n\nand no read ever follows. Second, if `cache_read_input_tokens`\n\nstays at zero across requests that look identical, something in the prefix is quietly changing between calls — a `datetime.now()`\n\nbaked into the system prompt, a `uuid4()`\n\nnear the front, `json.dumps(d)`\n\ncalled without `sort_keys=True`\n\n, or a tool list assembled from an unordered set.\n\nNot every call needs a response in two seconds. Nightly report summarization, bulk document classification, backfilling embeddings metadata, and re-scoring an eval set are not latency-sensitive — and that is exactly the workload profile the Message Batches API is priced for.\n\nAccording to the writeup, the Batches API discounts standard token pricing by **50%** in exchange for asynchronous processing: most batches finish within an hour, the hard ceiling is 24 hours, and results stay retrievable for 29 days. The two discounts stack independently — a batch of 10,000 classification calls that all share one large system prompt gets both the 50% batch discount and the cache-read discount on that shared prefix, per the writeup's own example.\n\nThe cheapest tokens are the ones sent to a cheaper model. In the writeup's support-ticket example, if 80% of tickets are confidently triaged by Haiku at $1/$5 per million tokens, and only the remaining 20% escalate to Opus at $5/$25 per million tokens, the blended cost comes out to a fraction of routing everything through Opus — with no quality loss on the easy majority, because the escalation path exists for exactly the cases where the cheap model reports it isn't sure.\n\nThe writeup flags one failure mode to guard against directly: a cheap model that is overconfident. Routing only works if the triage step actually escalates uncertain cases instead of guessing past them.\n\nThe pressure to have this lever ready isn't abstract. A separate report tracking API pricing across vendors describes a full-blown price war erupting across every major AI provider, one where the premium tier is shrinking fast enough that GPT-4 Turbo — still in production use at some enterprises — is now, in that report's words, 'laughably overpriced' compared to what's currently available. A routing layer that can point traffic at whichever tier actually fits the task is what turns that shift into savings instead of just noise.\n\nEach lever has a matching way to lose money. Caching a prefix that changes on every request pays the write premium with zero reads — worse than not caching at all. Sending latency-sensitive traffic through the Batches API breaks the product for users waiting on a synchronous reply. And routing decisions built on a cheap model's confidence score, without an escalation path, just moves the quality problem downstream instead of removing it.\n\nBefore implementing any of the three levers, pull the usage fields the API already returns on every response. They answer whether a lever will pay off before a line of routing logic gets written.\n\nCheckWhat to look atWhat it tells youCache is actually hitting`cache_read_input_tokens`\n\nvs. `cache_creation_input_tokens`\n\nZero reads means the prefix isn't stable — fix that before trusting the TTL mathPrefix size vs. minimumToken count of the cached prefixBelow the model's minimum, caching silently does nothingLatency tolerance per call siteIs a synchronous response requiredAnything that can wait an hour is a Batches API candidateTriage confidence distributionCheap-model confidence scores on real trafficSets the escalation threshold before routing goes live\n\n*This article was originally published on NextFuture. Follow us for more fullstack & AI engineering content.*", "url": "https://wpnews.pro/news/prompt-caching-batches-api-and-model-routing-to-cut-llm-costs", "canonical_source": "https://dev.to/bean_bean/prompt-caching-batches-api-and-model-routing-to-cut-llm-costs-3ak9", "published_at": "2026-07-13 11:00:01+00:00", "updated_at": "2026-07-13 11:15:13.710413+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Anthropic", "Claude", "Opus", "Haiku", "Python"], "alternates": {"html": "https://wpnews.pro/news/prompt-caching-batches-api-and-model-routing-to-cut-llm-costs", "markdown": "https://wpnews.pro/news/prompt-caching-batches-api-and-model-routing-to-cut-llm-costs.md", "text": "https://wpnews.pro/news/prompt-caching-batches-api-and-model-routing-to-cut-llm-costs.txt", "jsonld": "https://wpnews.pro/news/prompt-caching-batches-api-and-model-routing-to-cut-llm-costs.jsonld"}}