{"slug": "optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2", "title": "Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 2]", "summary": "A practical engineering playbook outlines three production techniques to reduce LLM token costs without sacrificing intelligence, with model routing as the first strategy. The approach uses a routing layer to match task complexity to the appropriate model, sending only a small percentage of requests to the most expensive model. The playbook argues that rising AI bills stem from many small architectural decisions rather than a single expensive model.", "body_md": "In [ Part 1](https://medium.com/ai-in-plain-english/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-1-26a41b7b1de5), we answered one important question:\n\n*Where do all the tokens actually come from and what are the cases that we actually ignore while building the AI application?*\n\nWe discovered that rising AI bills are rarely caused by a single expensive model. Instead, they’re usually the result of many small architectural decisions — long conversation histories, oversized prompts, repeated instructions, and unnecessary context.\n\nUnderstanding the problem is only the first step. The next challenge is far more practical.\n\n**How do we reduce token consumption without making the application less intelligent?**\n\nWhen developers first start optimizing LLM costs, the instinct is often to search for a cheaper model. While switching models can reduce expenses, it rarely addresses the root cause.\n\n*Imagine replacing a luxury car with an economy car while still driving hundreds of unnecessary kilometers every day. Your fuel bill might decrease, but you’re still wasting fuel.*\n\nThe same principle applies to LLM applications.\n\nBefore changing the model, we should first reduce the amount of unnecessary work we’re asking the model to perform.\n\nIn this article, we’ll explore three production techniques that consistently provide immediate savings:\n\nIndividually, these optimizations may seem small.\n\nTogether, they can dramatically reduce your AI bill while maintaining the same user experience. We will also see it with a hypothetical example.\n\nSooo…Let’s gooo….\n\nLet’s start with a simple question.\n\nSuppose your AI assistant receives the following requests within a few seconds:\n\n“Translate this paragraph into French.”\n\n“Summarize today’s meeting notes.”\n\n“Review this 300-line Python function and identify potential security vulnerabilities.”\n\nWould you process all three requests using the same language model?**Probably not.**\n\nThe first request requires very little reasoning.\n\nThe second requires moderate understanding.\n\nThe third demands much deeper reasoning and code comprehension.\n\nYet *many applications send every request to their most powerful — and most expensive — model*. It works, but it’s far from efficient.\n\nA better approach is to let the complexity of the task determine the model that handles it. This strategy is known as **Model Routing**.\n\nProduction AI applications rarely perform a single task. A customer support assistant may answer FAQs, summarize tickets, classify feedback, and generate detailed troubleshooting steps — all within the same application. Each task has different requirements.\n\nSome queries prioritize speed. Others prioritize reasoning. Some simply format text. Others analyze lengthy documents. So, there can be different models for different tasks.\n\nInstead of asking, *“Which is the best model?”*production teams usually ask:\n\nThat small change in perspective often leads to significant savings.\n\nA routing layer sits between your application and the language models. Rather than sending every request directly to an LLM, the application first decides **what kind of task** it has received.\n\nThe routing decision itself takes only a few milliseconds, but it can significantly reduce the number of expensive model invocations.\n\nFor tasks like *Translate an email -> *Developer can use* Small model.*Similarly,\n\nNotice that only a small percentage of requests genuinely require the most capable model. The remaining requests can often be handled by smaller, faster, and more affordable models with little or no impact on quality.\n\nRouting logic doesn’t have to be complicated. It can be through simple snippet :\n\n``` python\ndef route_request(task):    if task in [\"translation\", \"classification\"]:        return \"small-model\"    elif task == \"summarization\":        return \"medium-model\"    else:        return \"large-model\"\n```\n\nIn real production systems, routing may be based on classifiers, confidence scores, or historical performance rather than simple rules, but the underlying principle remains the same.\n\n**Ask a question by yourself** Think about your own application. If half of your requests involve summarization, formatting, or extraction, do they really need your most expensive model?\n\nOne of the biggest mistakes teams make is assuming thatevery request deserves the smartest model available.\n\nIn reality, most production traffic consists of repetitive, predictable tasks. Reserving premium models for only the most complex requests improves both cost efficiency and scalability.\n\nOnce you’ve ensured that requests are reaching the right model, the next question becomes:\n\n**Are we repeatedly sending the same information?**\n\nThis happens more often than many teams realize. Most production prompts contain instructions such as:\n\nThese instructions remain unchanged across thousands of requests. Yet they are transmitted to the model again and again.*Although users ask different questions, the application continues paying for the same instructions every single time.*This is exactly the problem that\n\nA prompt generally consists of two parts.**Static Content: **Information that rarely changes.\n\nExamples include:\n\n**Dynamic Content: **Information that changes with each request.\n\nExamples include:\n\nThe dynamic portion needs fresh processing. The static portion is an excellent candidate for caching.\n\nInstead of rebuilding the same prompt for every request, the application stores reusable sections and combines them with the latest user input.\n\nAlthough the optimization appears simple, eliminating repeated prompt construction can reduce millions of unnecessary tokens in high-traffic applications.\n\nIt can be implemented through simple code snippet:\n\n```\ncached_prompt = get_cached_prompt()final_prompt = cached_prompt + user_queryresponse = llm(final_prompt)\n```\n\nPrompt caching is most effective when your application sends large, reusable instructions with every request.\n\nCustomer support assistants, enterprise knowledge bots, and documentation search systems often benefit significantly because most of their prompts share the same foundation.\n\nHowever, *remember that caches must be refreshed whenever the underlying documentation or instructions change. Outdated cached prompts can lead to outdated responses.*\n\nBy now, we’ve optimized **which model handles each request** and **eliminated repeated prompt processing**. But there’s another hidden source of token growth that’s easy to overlook — **conversation history**.\n\nImagine you’re chatting with an AI assistant. Your first question might be:** User:** Summarize this report.\n\nAlthough your latest message is only a few words long, many applications send the **entire conversation** back to the model with every request. As the conversation grows, so does the prompt — and eventually, you’re paying more for old messages than for the user’s latest question.\n\nLarge Language Models don’t remember previous conversations. To maintain context, your application must send earlier messages along with the latest request. A typical request looks like this:\n\n*System Prompt + Conversation History + Current User Query -> LLM*\n\nThe problem is that conversation history keeps growing, even when much of it is no longer useful.\n\nInstead of sending every previous message, production systems often summarize older conversations while keeping the most recent exchanges intact. The workflow becomes much simpler.\n\n*Old Messages (becomes Conversation Summary) + Recent Messages + Current User Query -> LLM*\n\nThis allows the model to retain important context while significantly reducing the number of tokens sent with every request.\n\nSummarization doesn’t need to happen after every message. Most production systems trigger it only when:\n\nThis keeps the additional processing cost low while preventing conversations from growing indefinitely.\n\nWe can done through simple code snippet:\n\n```\nMAX_MESSAGES = 12if len(chat_history) > MAX_MESSAGES:    summary = summarize(chat_history[:-4])    chat_history = [        {\"role\": \"system\", \"content\": summary}    ] + chat_history[-4:]\n```\n\nHere, older messages are compressed into a summary, while the latest four messages remain unchanged.\n\nConversation summarization isn’t just about reducing costs — it also helps the model focus on what matters.\n\nLong conversations often contain greetings, repeated confirmations, and discussions that are no longer relevant. Replacing them with a concise summary keeps prompts smaller and responses more focused.\n\nA good rule of thumb is:Preserve the important information, not every single message.\n\nLet’s step back and look at our optimization journey so far. We started with an application that processed every request using the largest model, rebuilt the same prompts repeatedly, and resent the entire conversation history with every interaction.\n\nAfter introducing three relatively straightforward optimizations, the architecture looks much healthier. If we optimize our architecture with utmost optimization then we can reduce unnecessary cost by 70%-90%.\n\nWhat’s interesting is that none of these optimizations required changing the application’s core functionality. An hypothetical example of cost reduce using these techniques:\n\nUsers continued asking the same questions. The model continued producing high-quality responses. The engineering team simply reduced the amount of unnecessary work being sent to the model.\n\nBy now, we’ve optimized **how requests are created** before they reach the LLM. We learned that reducing costs isn’t always about choosing a cheaper model. Sometimes it’s about making better engineering decisions.\n\nIn this article, we explored three techniques that consistently deliver immediate production savings:\n\nTogether, these optimizations can reduce token usage dramatically while preserving the user experience.\n\nBut we’re not done yet — the picture is still pending.\n\nEven after optimizing requests, many applications still waste thousands of tokens retrieving information that the model never actually uses.\n\nIn **Part 3, **we’ll shift our focus from request optimization to **context optimization**. We’ll explore adaptive retrieval, batch inference, and output budgeting — three additional techniques that help production AI systems reduce costs, improve latency, and scale more efficiently.\n\nBecause in production AI, success isn’t measured by how many tokens you process — it’s measured by how few you need to achieve the same result.\n\nPart 3: [Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 2]](https://medium.com/@GarvitAgarwal/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-3-79556571e518)\n\n[Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 2]](https://pub.towardsai.net/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2-4f5e4d0eab53) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2", "canonical_source": "https://pub.towardsai.net/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2-4f5e4d0eab53?source=rss----98111c9905da---4", "published_at": "2026-07-21 03:42:48+00:00", "updated_at": "2026-07-21 04:29:45.431951+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-products"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2", "markdown": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2.md", "text": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2.txt", "jsonld": "https://wpnews.pro/news/optimizing-llm-token-costs-in-production-a-practical-engineering-playbook-part-2.jsonld"}}