Build a Semantic Cache for Your LLM App in 40 Lines of Python (And Cut Costs by Half)

A developer has published a guide to building a semantic cache for LLM applications in about 40 lines of Python, claiming it can cut costs by half. The approach uses sentence embeddings to match queries by meaning rather than exact strings, addressing a common failure mode where traditional caches miss paraphrased requests. The post also highlights a key pitfall—semantic similarity can conflate opposing intents like 'cancel' and 'reactivate'—and suggests a guard rail to mitigate it.

If you're calling an LLM API for every single user request, you're almost certainly paying for the same question more than once. Not because your users are dumb because human beings ask the same thing in a dozen different ways, and a normal cache only matches exact strings. "how do I reset my password" and "reset password help pls" are, to a dict or a Redis GET , completely unrelated. To your users, they're the same request. Every mismatch is a full-price model call you didn't need to make. Let's fix that. By the end of this post you'll have a working semantic cache, understand exactly where it breaks, and know how to fix that too. python import redis import hashlib r = redis.Redis def cached call prompt : key = hashlib.sha256 prompt.encode .hexdigest cached = r.get key if cached: return cached.decode response = call llm prompt expensive r.set key, response return response This works great in a demo. In production, your cache hit rate on real user traffic will hover somewhere near zero, because hashlib.sha256 doesn't know that "reset my password" and "help me reset password" mean the same thing. You've built a cache that only helps if someone copy-pastes the exact same request twice — which basically never happens with human-typed input. Instead of matching exact strings, we match meaning . We turn the query into a vector and compare it to vectors of queries we've already answered. python from sentence transformers import SentenceTransformer import numpy as np model = SentenceTransformer "all-MiniLM-L6-v2" fast, good enough for cache matching def embed text: str - np.ndarray: return model.encode text, normalize embeddings=True normalize embeddings=True matters — it means we can use a simple dot product as cosine similarity later, no extra normalization step needed. You don't need Qdrant or Milvus to learn this pattern you'll probably want one in production for scale more on that below . For a working prototype, an in-memory list is completely fine: python cache store = list of dicts: {embedding, query, response} def find match query embedding, threshold=0.92 : best score, best entry = 0, None for entry in cache store: score = float np.dot query embedding, entry "embedding" if score best score: best score, best entry = score, entry if best score threshold: return best entry return None php def semantic cached call prompt: str - str: query embedding = embed prompt match = find match query embedding if match: print f"cache hit score matched threshold " return match "response" response = call llm prompt your actual OpenAI/Anthropic call cache store.append { "embedding": query embedding, "query": prompt, "response": response, } return response That's the whole thing. Run it against a batch of paraphrased test queries and you'll immediately see cache hits on things a string cache would have completely missed. Try this: print semantic cached call "How do I cancel my subscription?" print semantic cached call "How do I reactivate my subscription?" Depending on your embedding model and threshold, there's a real chance the second query gets served the cancel answer, because "cancel" and "reactivate" share almost every other word in the sentence and embedding models cluster on shared vocabulary more than they distinguish directive verbs. This is the single most common failure mode in semantic caching, and it's why you can't ship the 40-line version above without a guard rail. Here's the fix a lightweight rejection rule for known polarity pairs, applied after the similarity match, before you trust it: OPPOSING PAIRS = "cancel", "reactivate" , "enable", "disable" , "add", "remove" , "increase", "decrease" , def has conflicting intent query: str, cached query: str - bool: q, c = query.lower , cached query.lower for a, b in OPPOSING PAIRS: if a in q and b in c or b in q and a in c : return True return False def safer semantic call prompt: str - str: query embedding = embed prompt match = find match query embedding if match and not has conflicting intent prompt, match "query" : return match "response" response = call llm prompt cache store.append {"embedding": query embedding, "query": prompt, "response": response} return response This won't catch every edge case — for that you'd want a proper cross-encoder rerank step or a small intent classifier — but it eliminates the single most embarrassing class of bug for almost no added complexity, and it's a good stopping point for a weekend project or an internal tool. The in-memory list works for a demo. For real traffic, swap it for an actual vector database the query pattern is the same, you're just trading for entry in cache store for an ANN index: python Using Qdrant as an example from qdrant client import QdrantClient from qdrant client.models import PointStruct, VectorParams, Distance client = QdrantClient ":memory:" swap for a real host in prod client.create collection collection name="semantic cache", vectors config=VectorParams size=384, distance=Distance.COSINE , def find match qdrant query embedding, threshold=0.92 : results = client.search collection name="semantic cache", query vector=query embedding.tolist , limit=1, if results and results 0 .score threshold: return results 0 .payload return None Same logic, now backed by an index that scales to millions of cached entries with sub-millisecond lookup. Mistake 1: One global threshold for every query type. A threshold tuned for "how do I use the API" is too loose for "what's my refund eligibility." If your app spans multiple domains, use different thresholds or exclude sensitive categories from caching entirely. Mistake 2: Caching everything, forever. Cached answers go stale the moment the underlying facts change pricing, policies, feature availability . Add a created at to every cache entry and expire aggressively for anything that isn't evergreen. Mistake 3: Trusting similarity score alone. As shown above, high similarity doesn't mean same intent. Always pair vector search with at least one lightweight guard rail before serving a cached response. Mistake 4: Not measuring hit rate by category. Overall hit rate is a vanity metric. A support bot might get 70% hits on "how do I..." questions and near 0% on account-specific ones know which is which so you can decide what's worth caching at all. Take the 40-line version above, hook it up to whatever LLM you're already calling, and log cache hits vs. misses for a day of real traffic. I'd genuinely bet you find at least a 30–40% hit rate on any app with repetitive user intents support, FAQs, internal tooling and every one of those hits is a full LLM call you just didn't have to pay for or wait on. If you want a more complete reference implementation with the guard-rail and invalidation logic built in, I put one together as an open-source package called Remem https://github.com/hrsvd/remem pip install remem-ai worth a look once you've built your own version and want to compare notes on the harder edge cases. Your turn: build the 40-line version, run it against your own logs or a paraphrased test set, and tell me your hit rate in the comments. I want to know if 30–40% holds up outside my own traffic patterns.