In Part 2, we focused on optimizing
Many applications continue to spend thousands — or even millions — of unnecessary tokens after the request has already been optimized.
How?
Because they still:
None of these problems originate from the language model itself. They’re engineering decisions.
And just like the techniques we discussed in Part 2, they can often be improved without changing models or sacrificing response quality.
Let’s look at three more production optimization techniques that help AI systems become faster, cheaper, and more scalable.
Retrieval-Augmented Generation (RAG) has become one of the most common architectures for production AI applications. Instead of relying solely on the model’s training data, a RAG pipeline retrieves relevant information from an external knowledge base before generating a response.
The idea is simple:
Give the model the right context so it can produce a more accurate answer.
The challenge is deciding how much context to retrieve.
Imagine you’re building an internal company chatbot.
A user asks: *“What are your office hours?”*Your vector database retrieves
documents = vectorstore.similarity_search(query, k=10)
Those documents might include: Employee handbook, HR policy, Security guidelines, Travel policy and so on. The answer only needs one sentence. Yet thousands of tokens are sent to the language model.
Now imagine this happens for 50,000 requests every day.
Most of those retrieved tokens contribute nothing to the final answer — but you’re still paying for them.
One Size Doesn’t Fit Every Query
Not every question deserves the same amount of context.
Compare these two requests.
**Query 1: ***“What are your office hours?”*A couple of relevant documents are enough.
Now consider, **Query 2: ***“Compare our healthcare reimbursement policy with last year’s finance guidelines.”*This question requires multiple documents from different sources.
Both queries are important. But they shouldn’t retrieve the same amount of information.
Making Retrieval Adaptive Instead of treating every query equally, production systems first estimate its complexity. Simple factual questions retrieve a small amount of context. Broader analytical questions retrieve more.
A simplified implementation looks like this:
if query_type == "simple": top_k = 2elif query_type == "medium": top_k = 5else: top_k = 10documents = vectorstore.similarity_search(query, k=top_k)p
The logic isn’t complicated. But over millions of requests, this small engineering decision can eliminate a huge amount of unnecessary token processing.
Reranking: Quality Matters More Than Quantity
Retrieving more documents doesn’t necessarily improve answer quality. Production systems often perform a second filtering step called reranking.
Instead of passing every retrieved document to the model:
Retriever — > Top 10 Documents — > Reranker — > Top 3 Relevant Documents — >LLM
The reranker scores each document according to its relevance and forwards only the best matches.
This has two advantages:First, the model processes fewer tokens.Second, it receives higher-quality context.
In many cases, fewer documents actually produce better responses because the model isn’t distracted by irrelevant information.
Many teams spend weeks experimenting with better embedding models.
Sometimes the biggest improvement comes from something much simpler:Stop sending documents the model doesn’t need.A smaller, cleaner context often improves bothaccuracyandcost efficiency.
So far, we’ve optimized what reaches the language model. But optimization isn’t just about reducing tokens. It’s also about how efficiently requests are processed.
This becomes particularly important when your AI application isn’t serving a single user — it might be processing thousands of documents, emails, product descriptions, or customer reviews every hour. At this scale, sending one request at a time can become surprisingly expensive.
Imagine you’re building a document search system. Before users can search your documents, each one needs to be converted into an embedding and stored in a vector database.
Suppose you have 1,000 PDF documents waiting to be indexed.
A straightforward implementation might look like this:
for document in documents: embedding = embedding_model.embed(document) vector_db.insert(embedding)
It works. But behind the scenes, you’re making 1,000 separate API calls.
Each request carries its own:
The model spends almost as much time handling requests as it does generating embeddings.
Instead of sending documents individually, production systems process them in batches.
batch_size = 100for i in range(0, len(documents), batch_size): batch = documents[i:i + batch_size] embeddings = embedding_model.embed(batch) vector_db.insert(embeddings)
Now, instead of making 1,000 API calls, you’re making only 10.
The number of tokens remains almost the same, but the infrastructure becomes far more efficient.
Think of ordering coffee for your team. Would you rather:
Both approaches produce the same result. One simply wastes much less time.
Batch inference works in exactly the same way.
Instead of repeatedly setting up new requests, the system processes multiple inputs together, reducing overhead and improving throughput.
Batching is most effective for workloads that don’t require an immediate response. Some common examples include:
These are background jobs where processing speed matters more than instant user interaction.
For real-time chatbots, however, batching is often less suitable because users expect responses within a few seconds.
One question we often ask while designing AI pipelines is:If the answer is“Does this request need to be processed immediately?”no, it’s usually a good candidate for batching.
This simple decision can improve throughput dramatically without changing the underlying AI model. Sometimes, optimizing an AI system isn’t about making the model faster. It’s about making thepipeline smarter.
By now, we’ve optimized how requests are built, how context is retrieved, and how efficiently requests are processed.
But there’s one final piece of the puzzle that many teams overlook:
The model’s response itself.
Every token generated by the LLM contributes to your bill. If your application consistently produces responses that are longer than necessary, you’re paying for information your users may never read.
Imagine you’re building an AI assistant for internal documentation.
A user asks: ** “What is vector search?”**The application only needs a brief explanation.
However, because the prompt doesn’t specify a desired response length, the model generates an 800-word article covering its history, algorithms, indexing strategies, and implementation details. Technically, the answer is excellent.
But did the user really need all of that?Probably not.
This is a common issue in production AI systems. Models naturally tend to provide detailed responses unless they’re given clear boundaries.
One of the simplest ways to control output cost is by limiting the maximum number of generated tokens.
For example:
response = client.responses.create( model="gpt-4.1-mini", input=prompt, max_output_tokens=200)
This doesn’t force the model to generate exactly 200 tokens. Instead, it prevents responses from growing unnecessarily long.Think of it as defining a budget.
Just as engineering teams monitor CPU, memory, and storage usage, it’s equally important to monitor how many output tokens your application generates.
Output budgeting isn’t only about token limits. It’s also about giving the model clearer instructions.
Consider these two prompts.
*Prompt 1: *** Explain Retrieval-Augmented Generation.
Now compare it with:
*Prompt 2: *** Explain Retrieval-Augmented Generation in five concise bullet points suitable for software engineers.
The second prompt communicates both the content and the expected format. As a result, the response is usually shorter, easier to understand, and consumes fewer output tokens.
Small improvements in prompt design often lead to noticeable savings when your application serves thousands of users.
When reviewing production AI systems, one question is surprisingly effective:
“Will the user benefit from a longer answer, or simply a clearer one?”
Most users aren’t looking for the longest response. They’re looking for the right response. By controlling output length and encouraging concise formatting, you not only reduce costs but also improve readability and user experience.
Sometimes, better AI isn’t about saying more. It’s about saying exactly what’s needed.
Across Part 2 and Part 3, we’ve explored six practical optimization techniques that production AI teams use every day.
Notice something interesting.
None of these techniques required training a new model. None required changing model providers. And none sacrificed response quality.
Instead, they focused on improving the engineering surrounding the model. That’s an important mindset shift.
The biggest improvements in production AI often come fromsystem design, not model selection.
Throughout this series, we’ve explored where LLM tokens come from and the engineering techniques that keep them under control. From model routing and prompt caching to adaptive retrieval, batch inference, and output budgeting, each optimization targets a different source of inefficiency.
Individually, these techniques may seem small. Together, they fundamentally change how production AI systems are designed.
The biggest lesson isn’t that language models are expensive.
It’s that unnecessary computation is expensive.
Every unnecessary document, repeated prompt, oversized response, or redundant request quietly adds to your operational costs. Remove those inefficiencies, and you don’t just reduce your AI bill — you build systems that are faster, more scalable, and easier to maintain.
The goal isn’t to process more tokens. It’s to make every token count.
Optimizing LLM Token Costs in Production: A Practical Engineering Playbook [Part 3] was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.