Building a semantic cache layer in front of RAG — and why it might be the most underrated cost optimization in production AI systems.
Hey Dev community 👋
Every RAG architecture diagram I see looks exactly the same.
User → Vector Search → LLM → Response.
But I keep wondering...
Why are we paying for the same answer thousands of times?
Imagine 50,000 customers asking "How do I block my debit card?" in slightly different words. Why should the system perform 50,000 vector searches and 50,000 LLM calls to generate 50,000 nearly-identical answers?
Maybe I'm missing something. Here's the architecture that's been stuck in my head — and I want people who've run RAG in production to tell me where it falls apart.
Try this thought experiment before you keep reading: imagine you're building a banking chatbot with one million customers. If 70% of customer questions repeat every day, would you really pay for a vector search and an LLM call every single time? Or should your semantic cache eventually become the first place you look?
Picture a bank's customer-facing chatbot that tries to resolve issues before a human agent gets involved.
Customer
│
▼
AI Chatbot
│
Answer?
│
Yes ──► Done ✅
│
No
│
▼
Human Agent
That chatbot does the same expensive thing under the hood for every question: vector search → LLM generation, every single time.
That's the piece I want to question.
Engineers care about numbers, not "thousands of users," so here's the scenario I'm imagining. These are illustrative assumptions, not measured data — I'm labeling them clearly so nobody mistakes this for a benchmark:
If 70% of 120,000 daily chats could be answered in 40ms instead of 2-4 seconds — and without a vector search or LLM call — that's not a minor latency win. That's a fundamentally different cost curve.
💡
HypothesisThe more customers use the chatbot,
the less the chatbot depends on RAG.
To state it more precisely: my hypothesis is that RAG gradually shifts from being the default execution path to handling only cache misses and newly emerging questions. Every cacheable question enriches the semantic cache. Every repeated question increases the cache hit rate. Over time, the system becomes less dependent on expensive retrieval.
Everything below is me trying to work out whether that hypothesis actually survives contact with production.
Here's the textbook RAG pipeline most of us ship first:
User Question
│
▼
Embed Question
│
▼
Vector Search (Pinecone / pgvector / Redis)
│
▼
Retrieve Top-K Chunks
│
▼
LLM (generate answer using context)
│
▼
Response
This works. It's also stateless in the worst way — the system has zero memory that it already answered this exact question 40 times today. Every repeat question pays the full vector-search-plus-LLM tax again.
For a support bot fielding thousands of near-duplicate questions a day — "How do I block my debit card?", "My ATM card is lost", "Can I disable my card online?" — that's a lot of wasted spend on semantically identical work.
Instead of a cache that only matches identical strings, what if we cache by meaning?
The first user pays the full RAG cost. Every similar question after that is an opportunity to avoid paying it again.
A semantic cache embeds the incoming question and compares it against previously answered questions using vector similarity. If the similarity score clears a threshold (say, 0.92
cosine similarity), it returns the cached answer — no vector search against the full knowledge base, no LLM call.
A normal Redis cache only works when two requests are exactly identical:
"How do I block my debit card?"
!=
"My ATM card is lost."
!=
"Disable my card."
Three different strings, three different cache keys, three cache misses — even though every one of them wants the same answer. A normal cache has no concept of meaning, only exact matches.
A semantic cache recognizes that all three express the same intent. That's why embeddings matter — they let the system compare questions by what they mean, not by how they're typed.
A normal cache would treat these as three different keys:
A semantic cache collapses all three into one lookup:
┌───────────────────────────┐
│ Semantic Cache │
│ (Redis + Embeddings) │
└─────────────┬───────────────┘
│
similarity ≥ threshold?
┌───────────┴───────────┐
YES NO
│ │
▼ ▼
Return Cached Answer RAG Pipeline
(milliseconds) │
Vector Search + LLM
│
▼
Store {embedding, answer} in Redis
│
▼
Return Response
The key shift: in this proposed architecture, RAG becomes the fallback path rather than the default.
This is where the idea breaks if you're not careful, so let's draw the line early.
Safe to cache — general knowledge-base questions with stable answers:
Never cache — anything tied to live, personal, or account-specific state:
Incoming Question
│
▼
Is this a policy/FAQ question, or does it need live account data?
│
┌────┴─────┐
POLICY/FAQ LIVE DATA
│ │
▼ ▼
Semantic Always hit live
Cache systems directly
eligible (never cached)
In practice, this usually means classifying intent before the cache lookup — a lightweight router that decides "cacheable knowledge query" vs. "personalized live query," and only sends the former through the semantic cache.
I'm deliberately not dropping 70 lines of Redis client code here — this article isn't about the API calls, it's about the architecture and the traffic pattern.
In practice, this could be built with Redis Vector Search, pgvector, or any other embedding store: embed the incoming question, run a KNN similarity lookup, return the cached answer above a threshold, otherwise fall through to the RAG pipeline and write the new answer back to the cache. The implementation isn't the interesting part — whether the traffic pattern I'm describing actually holds up is.
One thing I'd expect any real version of this to need: in production, I'd expect only validated or high-confidence answers to be written back into the semantic cache — not every RAG output. A hallucinated or low-confidence response getting cached and reused thousands of times would be worse than generating it fresh every time.
Maybe RAG isn't supposed to answer every question forever.
Maybe its real job is to answer new questions.
Once an answer proves useful and keeps getting requested — why shouldn't it graduate into the semantic cache?
Today
100 requests
│
▼
100 RAG calls
6 months later
100 requests
│
▼
80 Cache hits
20 RAG calls
If that's the right way to think about it, RAG's job quietly changes over time — from "answer everything" to "answer the unknown."
I want to be upfront: I have no real measurements for this.
This is NOT production data. It's the mental model I'm trying to validate.
Here's the shape I'd expect if 80% of a support system's questions are repetitive and 20% are genuinely new:
Requests
100% |\
90% | \
80% | \
70% | \___
60% | \___ Expensive RAG requests
50% | \___
40% | \________
30% | ___________---- Semantic Cache hits
20% | ____/
10% |___/
0% |________________________________________ Time
Day 1 Week 1 Week 2 Month 1 Month 2 Month 3
| Time | RAG traffic | Cache hits |
|---|---|---|
| Day 1 | 100% | 0% |
| Week 1 | 85% | 15% |
| Week 2 | 70% | 30% |
| Month 1 | 55% | 45% |
| Month 2 | 40% | 60% |
| Month 3 | 25% | 75% |
If even half of this hypothesis holds, a mature semantic cache could become the primary retrieval layer for repetitive knowledge queries, with RAG handling only new, rare, or edge-case questions — cache invalidation, TTL expiry, and changing policies permitting.
Here's the question I can't answer:
If your cache hit rate eventually reaches 80%, is your semantic cache now your primary retrieval engine — and is RAG simply handling cache misses?
If that's a flawed way to think about production RAG, I'd genuinely love to know why.
Looking forward to the discussion