In Part 1, we answered one important question:
Where do all the tokens actually come from and what are the cases that we actually ignore while building the AI application?
We discovered that rising AI bills are rarely caused by a single expensive model. Instead, they’re usually the result of many small architectural decisions — long conversation histories, oversized prompts, repeated instructions, and unnecessary context.
Understanding the problem is only the first step. The next challenge is far more practical.
How do we reduce token consumption without making the application less intelligent?
When developers first start optimizing LLM costs, the instinct is often to search for a cheaper model. While switching models can reduce expenses, it rarely addresses the root cause.
Imagine replacing a luxury car with an economy car while still driving hundreds of unnecessary kilometers every day. Your fuel bill might decrease, but you’re still wasting fuel.
The same principle applies to LLM applications.
Before changing the model, we should first reduce the amount of unnecessary work we’re asking the model to perform.
In this article, we’ll explore three production techniques that consistently provide immediate savings:
Individually, these optimizations may seem small.
Together, they can dramatically reduce your AI bill while maintaining the same user experience. We will also see it with a hypothetical example.
Sooo…Let’s gooo….
Let’s start with a simple question.
Suppose your AI assistant receives the following requests within a few seconds:
“Translate this paragraph into French.”
“Summarize today’s meeting notes.”
“Review this 300-line Python function and identify potential security vulnerabilities.”
Would you process all three requests using the same language model?Probably not.
The first request requires very little reasoning.
The second requires moderate understanding.
The third demands much deeper reasoning and code comprehension.
Yet many applications send every request to their most powerful — and most expensive — model. It works, but it’s far from efficient.
A better approach is to let the complexity of the task determine the model that handles it. This strategy is known as Model Routing.
Production AI applications rarely perform a single task. A customer support assistant may answer FAQs, summarize tickets, classify feedback, and generate detailed troubleshooting steps — all within the same application. Each task has different requirements.
Some queries prioritize speed. Others prioritize reasoning. Some simply format text. Others analyze lengthy documents. So, there can be different models for different tasks.
Instead of asking, *“Which is the best model?”*production teams usually ask:
That small change in perspective often leads to significant savings.
A routing layer sits between your application and the language models. Rather than sending every request directly to an LLM, the application first decides what kind of task it has received.
The routing decision itself takes only a few milliseconds, but it can significantly reduce the number of expensive model invocations.
For tasks like *Translate an email -> Developer can use Small model.*Similarly,
Notice that only a small percentage of requests genuinely require the most capable model. The remaining requests can often be handled by smaller, faster, and more affordable models with little or no impact on quality.
Routing logic doesn’t have to be complicated. It can be through simple snippet :
def route_request(task): if task in ["translation", "classification"]: return "small-model" elif task == "summarization": return "medium-model" else: return "large-model"
In real production systems, routing may be based on classifiers, confidence scores, or historical performance rather than simple rules, but the underlying principle remains the same.
Ask a question by yourself Think about your own application. If half of your requests involve summarization, formatting, or extraction, do they really need your most expensive model?
One of the biggest mistakes teams make is assuming thatevery request deserves the smartest model available.
In reality, most production traffic consists of repetitive, predictable tasks. Reserving premium models for only the most complex requests improves both cost efficiency and scalability.
Once you’ve ensured that requests are reaching the right model, the next question becomes:
Are we repeatedly sending the same information?
This happens more often than many teams realize. Most production prompts contain instructions such as:
These instructions remain unchanged across thousands of requests. Yet they are transmitted to the model again and again.*Although users ask different questions, the application continues paying for the same instructions every single time.*This is exactly the problem that
A prompt generally consists of two parts.**Static Content: **Information that rarely changes.
Examples include:
**Dynamic Content: **Information that changes with each request.
Examples include:
The dynamic portion needs fresh processing. The static portion is an excellent candidate for caching.
Instead of rebuilding the same prompt for every request, the application stores reusable sections and combines them with the latest user input.
Although the optimization appears simple, eliminating repeated prompt construction can reduce millions of unnecessary tokens in high-traffic applications.
It can be implemented through simple code snippet:
cached_prompt = get_cached_prompt()final_prompt = cached_prompt + user_queryresponse = llm(final_prompt)
Prompt caching is most effective when your application sends large, reusable instructions with every request.
Customer support assistants, enterprise knowledge bots, and documentation search systems often benefit significantly because most of their prompts share the same foundation.
However, remember that caches must be refreshed whenever the underlying documentation or instructions change. Outdated cached prompts can lead to outdated responses.
By now, we’ve optimized which model handles each request and eliminated repeated prompt processing. But there’s another hidden source of token growth that’s easy to overlook — conversation history.
Imagine you’re chatting with an AI assistant. Your first question might be:** User:** Summarize this report.
Although your latest message is only a few words long, many applications send the entire conversation back to the model with every request. As the conversation grows, so does the prompt — and eventually, you’re paying more for old messages than for the user’s latest question.
Large Language Models don’t remember previous conversations. To maintain context, your application must send earlier messages along with the latest request. A typical request looks like this:
System Prompt + Conversation History + Current User Query -> LLM
The problem is that conversation history keeps growing, even when much of it is no longer useful.
Instead of sending every previous message, production systems often summarize older conversations while keeping the most recent exchanges intact. The workflow becomes much simpler.
Old Messages (becomes Conversation Summary) + Recent Messages + Current User Query -> LLM
This allows the model to retain important context while significantly reducing the number of tokens sent with every request.
Summarization doesn’t need to happen after every message. Most production systems trigger it only when:
This keeps the additional processing cost low while preventing conversations from growing indefinitely.
We can done through simple code snippet:
MAX_MESSAGES = 12if len(chat_history) > MAX_MESSAGES: summary = summarize(chat_history[:-4]) chat_history = [ {"role": "system", "content": summary} ] + chat_history[-4:]
Here, older messages are compressed into a summary, while the latest four messages remain unchanged.
Conversation summarization isn’t just about reducing costs — it also helps the model focus on what matters.
Long conversations often contain greetings, repeated confirmations, and discussions that are no longer relevant. Replacing them with a concise summary keeps prompts smaller and responses more focused.
A good rule of thumb is:Preserve the important information, not every single message.
Let’s step back and look at our optimization journey so far. We started with an application that processed every request using the largest model, rebuilt the same prompts repeatedly, and resent the entire conversation history with every interaction.
After introducing three relatively straightforward optimizations, the architecture looks much healthier. If we optimize our architecture with utmost optimization then we can reduce unnecessary cost by 70%-90%.
What’s interesting is that none of these optimizations required changing the application’s core functionality. An hypothetical example of cost reduce using these techniques:
Users continued asking the same questions. The model continued producing high-quality responses. The engineering team simply reduced the amount of unnecessary work being sent to the model.
By now, we’ve optimized how requests are created before they reach the LLM. We learned that reducing costs isn’t always about choosing a cheaper model. Sometimes it’s about making better engineering decisions.
In this article, we explored three techniques that consistently deliver immediate production savings:
Together, these optimizations can reduce token usage dramatically while preserving the user experience.
But we’re not done yet — the picture is still pending.
Even after optimizing requests, many applications still waste thousands of tokens retrieving information that the model never actually uses.
In **Part 3, **we’ll shift our focus from request optimization to context optimization. We’ll explore adaptive retrieval, batch inference, and output budgeting — three additional techniques that help production AI systems reduce costs, improve latency, and scale more efficiently.
Because in production AI, success isn’t measured by how many tokens you process — it’s measured by how few you need to achieve the same result.
Part 3: Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 2]
Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 2] was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.