If you're calling an LLM API for every single user request, you're almost certainly paying for the same question more than once. Not because your users are dumb because human beings ask the same thing in a dozen different ways, and a normal cache only matches exact strings.
"how do I reset my password" and "reset password help pls" are, to a dict
or a Redis GET
, completely unrelated. To your users, they're the same request. Every mismatch is a full-price model call you didn't need to make.
Let's fix that. By the end of this post you'll have a working semantic cache, understand exactly where it breaks, and know how to fix that too.
import redis
import hashlib
r = redis.Redis()
def cached_call(prompt):
key = hashlib.sha256(prompt.encode()).hexdigest()
cached = r.get(key)
if cached:
return cached.decode()
response = call_llm(prompt) # expensive
r.set(key, response)
return response
This works great in a demo. In production, your cache hit rate on real user traffic will hover somewhere near zero, because hashlib.sha256
doesn't know that "reset my password" and "help me reset password" mean the same thing. You've built a cache that only helps if someone copy-pastes the exact same request twice — which basically never happens with human-typed input.
Instead of matching exact strings, we match meaning. We turn the query into a vector and compare it to vectors of queries we've already answered.
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("all-MiniLM-L6-v2") # fast, good enough for cache matching
def embed(text: str) -> np.ndarray:
return model.encode(text, normalize_embeddings=True)
normalize_embeddings=True
matters — it means we can use a simple dot product as cosine similarity later, no extra normalization step needed.
You don't need Qdrant or Milvus to learn this pattern (you'll probably want one in production for scale more on that below). For a working prototype, an in-memory list is completely fine:
cache_store = [] # list of dicts: {embedding, query, response}
def find_match(query_embedding, threshold=0.92):
best_score, best_entry = 0, None
for entry in cache_store:
score = float(np.dot(query_embedding, entry["embedding"]))
if score > best_score:
best_score, best_entry = score, entry
if best_score > threshold:
return best_entry
return None
php
def semantic_cached_call(prompt: str) -> str:
query_embedding = embed(prompt)
match = find_match(query_embedding)
if match:
print(f"cache hit (score matched threshold)")
return match["response"]
response = call_llm(prompt) # your actual OpenAI/Anthropic call
cache_store.append({
"embedding": query_embedding,
"query": prompt,
"response": response,
})
return response
That's the whole thing. Run it against a batch of paraphrased test queries and you'll immediately see cache hits on things a string cache would have completely missed.
Try this:
print(semantic_cached_call("How do I cancel my subscription?"))
print(semantic_cached_call("How do I reactivate my subscription?"))
Depending on your embedding model and threshold, there's a real chance the second query gets served the cancel answer, because "cancel" and "reactivate" share almost every other word in the sentence and embedding models cluster on shared vocabulary more than they distinguish directive verbs. This is the single most common failure mode in semantic caching, and it's why you can't ship the 40-line version above without a guard rail.
Here's the fix a lightweight rejection rule for known polarity pairs, applied after the similarity match, before you trust it:
OPPOSING_PAIRS = [
("cancel", "reactivate"), ("enable", "disable"),
("add", "remove"), ("increase", "decrease"),
]
def has_conflicting_intent(query: str, cached_query: str) -> bool:
q, c = query.lower(), cached_query.lower()
for a, b in OPPOSING_PAIRS:
if (a in q and b in c) or (b in q and a in c):
return True
return False
def safer_semantic_call(prompt: str) -> str:
query_embedding = embed(prompt)
match = find_match(query_embedding)
if match and not has_conflicting_intent(prompt, match["query"]):
return match["response"]
response = call_llm(prompt)
cache_store.append({"embedding": query_embedding, "query": prompt, "response": response})
return response
This won't catch every edge case — for that you'd want a proper cross-encoder rerank step or a small intent classifier — but it eliminates the single most embarrassing class of bug for almost no added complexity, and it's a good stopping point for a weekend project or an internal tool.
The in-memory list works for a demo. For real traffic, swap it for an actual vector database the query pattern is the same, you're just trading for entry in cache_store
for an ANN index:
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance
client = QdrantClient(":memory:") # swap for a real host in prod
client.create_collection(
collection_name="semantic_cache",
vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)
def find_match_qdrant(query_embedding, threshold=0.92):
results = client.search(
collection_name="semantic_cache",
query_vector=query_embedding.tolist(),
limit=1,
)
if results and results[0].score > threshold:
return results[0].payload
return None
Same logic, now backed by an index that scales to millions of cached entries with sub-millisecond lookup.
Mistake 1: One global threshold for every query type. A threshold tuned for "how do I use the API" is too loose for "what's my refund eligibility." If your app spans multiple domains, use different thresholds or exclude sensitive categories from caching entirely.
Mistake 2: Caching everything, forever. Cached answers go stale the moment the underlying facts change (pricing, policies, feature availability). Add a created_at
to every cache entry and expire aggressively for anything that isn't evergreen.
Mistake 3: Trusting similarity score alone. As shown above, high similarity doesn't mean same intent. Always pair vector search with at least one lightweight guard rail before serving a cached response.
Mistake 4: Not measuring hit rate by category. Overall hit rate is a vanity metric. A support bot might get 70% hits on "how do I..." questions and near 0% on account-specific ones know which is which so you can decide what's worth caching at all.
Take the 40-line version above, hook it up to whatever LLM you're already calling, and log cache hits vs. misses for a day of real traffic. I'd genuinely bet you find at least a 30–40% hit rate on any app with repetitive user intents (support, FAQs, internal tooling) and every one of those hits is a full LLM call you just didn't have to pay for or wait on.
If you want a more complete reference implementation with the guard-rail and invalidation logic built in, I put one together as an open-source package called Remem (pip install remem-ai
) worth a look once you've built your own version and want to compare notes on the harder edge cases.
Your turn: build the 40-line version, run it against your own logs or a paraphrased test set, and tell me your hit rate in the comments. I want to know if 30–40% holds up outside my own traffic patterns.