{"slug": "til-six-ways-to-cut-llm-costs", "title": "TIL: Six ways to cut LLM costs", "summary": "A TIL post outlines six techniques for reducing large language model inference costs: prompt caching, model routing, batch APIs, semantic caching, context trimming and smaller or distilled models. The post cites Anthropic's Claude Platform documentation for prompt caching and batch processing, Redis documentation for semantic caching, and Hugging Face documentation for knowledge distillation, and advises teams to measure cost per accepted result by counting inference, routing, cache infrastructure, retries and human checking. It notes that quantisation reduces numerical precision rather than parameter count, and that changing a prompt prefix can cost prompt-cache hits.", "body_md": "# TIL: Six ways to cut LLM costs\n\n1. **Prompt caching.** Put stable instructions, examples and tool schemas before changing inputs. An identical prefix can reuse earlier computation; the model still generates a fresh answer. Measure cache hits and account for cache-write costs.[Claude Platform Docs](https://platform.claude.com/docs/en/build-with-claude/prompt-caching?ref=anup.io)\n2. **Model routing.** Send each task to the cheapest model that passes your evals. Use a larger model where it earns its cost. Include the router and fallback calls when measuring savings.\n3. **Batch API.** Move work that can wait, such as overnight extraction or offline evals, to an asynchronous batch API. Providers such as Anthropic discount these requests. You are trading response time for price.[Claude Platform Docs](https://platform.claude.com/docs/en/build-with-claude/batch-processing?ref=anup.io)\n4. **Semantic caching.** Reuse a stored answer when a new query has equivalent meaning and relevant context. A valid hit skips generation. Similar wording alone is insufficient: enforce access boundaries and invalidate stale answers.[Docs](https://redis.io/docs/latest/develop/use-cases/semantic-cache/?ref=anup.io)\n5. **Trim context.** Remove irrelevant retrieved chunks and bulky old tool results. Preserve task constraints, decisions and current state. Run evals after trimming, and watch for lost prompt-cache hits when you change the prefix.\n6. **Smaller models.** Choose a smaller model that meets the quality bar, or train one through distillation. Quantisation is different: it reduces numerical precision, not parameter count. For self-hosting, benchmark memory, throughput and quality on your actual hardware.[Hugging Face](https://huggingface.co/docs/transformers/main/tasks/knowledge_distillation_for_image_classification?ref=anup.io)\n\n**Measure cost per accepted result.** Count inference, routing, cache infrastructure, retries and human checking. Keep quality and latency visible alongside the bill.\n\n                                    Join AI engineers getting weekly insights on agents, RAG & production LLM systems\n                            \n\nNo spam, no sharing to third party. Only you and me.", "url": "https://wpnews.pro/news/til-six-ways-to-cut-llm-costs", "canonical_source": "https://www.anup.io/til-six-ways-to-cut-llm-costs/", "published_at": "2026-11-04 15:30:47+00:00", "updated_at": "2026-11-04 15:49:57.610109+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["Anthropic", "Claude Platform", "Redis", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/til-six-ways-to-cut-llm-costs", "markdown": "https://wpnews.pro/news/til-six-ways-to-cut-llm-costs.md", "text": "https://wpnews.pro/news/til-six-ways-to-cut-llm-costs.txt", "jsonld": "https://wpnews.pro/news/til-six-ways-to-cut-llm-costs.jsonld"}}