{"slug": "optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3", "title": "Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 3]", "summary": "A new engineering playbook details three production techniques to cut unnecessary LLM token costs after request optimization: adaptive retrieval that varies context size by query complexity, reranking to pass only the most relevant documents to the model, and efficient request processing for high-volume systems. The author, Garvit Agarwal, notes that these engineering decisions can eliminate millions of wasted tokens without changing models or sacrificing response quality.", "body_md": "In [ Part 2](https://medium.com/@GarvitAgarwal/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2-4f5e4d0eab53), we focused on optimizing\n\nMany applications continue to spend thousands — or even millions — of unnecessary tokens **after** the request has already been optimized.\n\n**How?**\n\nBecause they still:\n\nNone of these problems originate from the language model itself. They’re engineering decisions.\n\nAnd just like the techniques we discussed in Part 2, they can often be improved without changing models or sacrificing response quality.\n\nLet’s look at three more production optimization techniques that help AI systems become faster, cheaper, and more scalable.\n\nRetrieval-Augmented Generation (RAG) has become one of the most common architectures for production AI applications. Instead of relying solely on the model’s training data, a RAG pipeline retrieves relevant information from an external knowledge base before generating a response.\n\nThe idea is simple:\n\nGive the model the right context so it can produce a more accurate answer.\n\nThe challenge is deciding **how much** context to retrieve.\n\nImagine you’re building an internal company chatbot.\n\nA user asks: *“What are your office hours?”*Your vector database retrieves\n\n```\ndocuments = vectorstore.similarity_search(query, k=10)\n```\n\nThose documents might include: Employee handbook, HR policy, Security guidelines, Travel policy and so on. The answer only needs one sentence. Yet thousands of tokens are sent to the language model.\n\nNow imagine this happens for **50,000 requests every day**.\n\nMost of those retrieved tokens contribute nothing to the final answer — but you’re still paying for them.\n\n**One Size Doesn’t Fit Every Query**\n\nNot every question deserves the same amount of context.\n\nCompare these two requests.\n\n**Query 1: ***“What are your office hours?”*A couple of relevant documents are enough.\n\nNow consider, **Query 2: ***“Compare our healthcare reimbursement policy with last year’s finance guidelines.”*This question requires multiple documents from different sources.\n\nBoth queries are important. But they shouldn’t retrieve the same amount of information.\n\n**Making Retrieval Adaptive** Instead of treating every query equally, production systems first estimate its complexity. Simple factual questions retrieve a small amount of context. Broader analytical questions retrieve more.\n\nA simplified implementation looks like this:\n\n```\nif query_type == \"simple\":    top_k = 2elif query_type == \"medium\":    top_k = 5else:    top_k = 10documents = vectorstore.similarity_search(query, k=top_k)p\n```\n\nThe logic isn’t complicated. But over millions of requests, this small engineering decision can eliminate a huge amount of unnecessary token processing.\n\n**Reranking: Quality Matters More Than Quantity**\n\nRetrieving more documents doesn’t necessarily improve answer quality. Production systems often perform a second filtering step called **reranking**.\n\nInstead of passing every retrieved document to the model:\n\nRetriever — > Top 10 Documents — > Reranker — > Top 3 Relevant Documents — >LLM\n\nThe reranker scores each document according to its relevance and forwards only the best matches.\n\nThis has two advantages:*First, the model processes fewer tokens.Second, it receives higher-quality context.*\n\nIn many cases, fewer documents actually produce better responses because the model isn’t distracted by irrelevant information.\n\nMany teams spend weeks experimenting with better embedding models.\n\nSometimes the biggest improvement comes from something much simpler:Stop sending documents the model doesn’t need.A smaller, cleaner context often improves bothaccuracyandcost efficiency.\n\nSo far, we’ve optimized **what** reaches the language model. But optimization isn’t just about reducing tokens. It’s also about **how efficiently requests are processed**.\n\nThis becomes particularly important when your AI application isn’t serving a single user — it might be processing thousands of documents, emails, product descriptions, or customer reviews every hour. At this scale, sending one request at a time can become surprisingly expensive.\n\nImagine you’re building a document search system. Before users can search your documents, each one needs to be converted into an embedding and stored in a vector database.\n\nSuppose you have **1,000 PDF documents** waiting to be indexed.\n\nA straightforward implementation might look like this:\n\n```\nfor document in documents:    embedding = embedding_model.embed(document)    vector_db.insert(embedding)\n```\n\nIt works. But behind the scenes, you’re making **1,000 separate API calls**.\n\n*Each request carries its own:*\n\nThe model spends almost as much time handling requests as it does generating embeddings.\n\nInstead of sending documents individually, production systems process them in **batches**.\n\n```\nbatch_size = 100for i in range(0, len(documents), batch_size):    batch = documents[i:i + batch_size]    embeddings = embedding_model.embed(batch)    vector_db.insert(embeddings)\n```\n\nNow, instead of making **1,000 API calls**, you’re making only **10**.\n\nThe number of tokens remains almost the same, but the infrastructure becomes far more efficient.\n\n**Think of ordering coffee for your team.** Would you rather:\n\nBoth approaches produce the same result. One simply wastes much less time.\n\nBatch inference works in exactly the same way.\n\nInstead of repeatedly setting up new requests, the system processes multiple inputs together, reducing overhead and improving throughput.\n\nBatching is most effective for workloads that don’t require an immediate response. Some common examples include:\n\nThese are background jobs where processing speed matters more than instant user interaction.\n\nFor real-time chatbots, however, batching is often less suitable because users expect responses within a few seconds.\n\nOne question we often ask while designing AI pipelines is:If the answer is“Does this request need to be processed immediately?”no, it’s usually a good candidate for batching.\n\nThis simple decision can improve throughput dramatically without changing the underlying AI model. Sometimes, optimizing an AI system isn’t about making the model faster. It’s about making thepipeline smarter.\n\nBy now, we’ve optimized how requests are built, how context is retrieved, and how efficiently requests are processed.\n\nBut there’s one final piece of the puzzle that many teams overlook:\n\n**The model’s response itself.**\n\nEvery token generated by the LLM contributes to your bill. If your application consistently produces responses that are longer than necessary, you’re paying for information your users may never read.\n\nImagine you’re building an AI assistant for internal documentation.\n\nA user asks: ** “What is vector search?”**The application only needs a brief explanation.\n\nHowever, because the prompt doesn’t specify a desired response length, the model generates an 800-word article covering its history, algorithms, indexing strategies, and implementation details. Technically, the answer is excellent.\n\nBut did the user really need all of that?**Probably not.**\n\nThis is a common issue in production AI systems. Models naturally tend to provide detailed responses unless they’re given clear boundaries.\n\nOne of the simplest ways to control output cost is by limiting the maximum number of generated tokens.\n\nFor example:\n\n```\nresponse = client.responses.create(    model=\"gpt-4.1-mini\",    input=prompt,    max_output_tokens=200)\n```\n\nThis doesn’t force the model to generate exactly 200 tokens. Instead, it prevents responses from growing unnecessarily long.Think of it as defining a budget.\n\n*Just as engineering teams monitor CPU, memory, and storage usage, it’s equally important to monitor how many output tokens your application generates.*\n\nOutput budgeting isn’t only about token limits. It’s also about giving the model clearer instructions.\n\nConsider these two prompts.\n\n**Prompt 1: *** Explain Retrieval-Augmented Generation.*\n\nNow compare it with:\n\n**Prompt 2: *** Explain Retrieval-Augmented Generation in five concise bullet points suitable for software engineers.*\n\nThe second prompt communicates both the content and the expected format. As a result, the response is usually shorter, easier to understand, and consumes fewer output tokens.\n\n*Small improvements in prompt design often lead to noticeable savings when your application serves thousands of users.*\n\nWhen reviewing production AI systems, one question is surprisingly effective:\n\n“Will the user benefit from a longer answer, or simply a clearer one?”\n\nMost users aren’t looking for the longest response. They’re looking for the right response. By controlling output length and encouraging concise formatting, you not only reduce costs but also improve readability and user experience.\n\nSometimes, better AI isn’t about saying more. It’s about saying exactly what’s needed.\n\nAcross **Part 2** and **Part 3**, we’ve explored six practical optimization techniques that production AI teams use every day.\n\nNotice something interesting.\n\n*None of these techniques required training a new model. None required changing model providers. And none sacrificed response quality.*\n\nInstead, they focused on improving the engineering surrounding the model. That’s an important mindset shift.\n\nThe biggest improvements in production AI often come fromsystem design, not model selection.\n\nThroughout this series, we’ve explored where LLM tokens come from and the engineering techniques that keep them under control. From **model routing** and **prompt caching** to **adaptive retrieval**, **batch inference**, and **output budgeting**, each optimization targets a different source of inefficiency.\n\nIndividually, these techniques may seem small. Together, they fundamentally change how production AI systems are designed.\n\nThe biggest lesson isn’t that language models are expensive.\n\nIt’s that **unnecessary computation is expensive.**\n\nEvery unnecessary document, repeated prompt, oversized response, or redundant request quietly adds to your operational costs. Remove those inefficiencies, and you don’t just reduce your AI bill — you build systems that are faster, more scalable, and easier to maintain.\n\n*The goal isn’t to process more tokens. It’s to make every token count.*\n\n[Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 3]](https://pub.towardsai.net/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3-79556571e518) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3", "canonical_source": "https://pub.towardsai.net/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3-79556571e518?source=rss----98111c9905da---4", "published_at": "2026-07-20 05:53:04+00:00", "updated_at": "2026-07-20 13:22:09.321624+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-infrastructure", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3", "markdown": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3.md", "text": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3.txt", "jsonld": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3.jsonld"}}