{"slug": "build-a-semantic-cache-for-your-llm-app-in-40-lines-of-python-and-cut-costs-by", "title": "Build a Semantic Cache for Your LLM App in 40 Lines of Python (And Cut Costs by Half)", "summary": "A developer has published a guide to building a semantic cache for LLM applications in about 40 lines of Python, claiming it can cut costs by half. The approach uses sentence embeddings to match queries by meaning rather than exact strings, addressing a common failure mode where traditional caches miss paraphrased requests. The post also highlights a key pitfall—semantic similarity can conflate opposing intents like 'cancel' and 'reactivate'—and suggests a guard rail to mitigate it.", "body_md": "If you're calling an LLM API for every single user request, you're almost certainly paying for the same question more than once. Not because your users are dumb because human beings ask the same thing in a dozen different ways, and a normal cache only matches exact strings.\n\n\"how do I reset my password\" and \"reset password help pls\" are, to a `dict`\n\nor a Redis `GET`\n\n, completely unrelated. To your users, they're the same request. Every mismatch is a full-price model call you didn't need to make.\n\nLet's fix that. By the end of this post you'll have a working semantic cache, understand exactly where it breaks, and know how to fix that too.\n\n``` python\nimport redis\nimport hashlib\n\nr = redis.Redis()\n\ndef cached_call(prompt):\n    key = hashlib.sha256(prompt.encode()).hexdigest()\n    cached = r.get(key)\n    if cached:\n        return cached.decode()\n    response = call_llm(prompt)  # expensive\n    r.set(key, response)\n    return response\n```\n\nThis works great in a demo. In production, your cache hit rate on real user traffic will hover somewhere near zero, because `hashlib.sha256`\n\ndoesn't know that \"reset my password\" and \"help me reset password\" mean the same thing. You've built a cache that only helps if someone copy-pastes the exact same request twice — which basically never happens with human-typed input.\n\nInstead of matching exact strings, we match *meaning*. We turn the query into a vector and compare it to vectors of queries we've already answered.\n\n``` python\nfrom sentence_transformers import SentenceTransformer\nimport numpy as np\n\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")  # fast, good enough for cache matching\n\ndef embed(text: str) -> np.ndarray:\n    return model.encode(text, normalize_embeddings=True)\n```\n\n`normalize_embeddings=True`\n\nmatters — it means we can use a simple dot product as cosine similarity later, no extra normalization step needed.\n\nYou don't need Qdrant or Milvus to *learn* this pattern (you'll probably want one in production for scale more on that below). For a working prototype, an in-memory list is completely fine:\n\n``` python\ncache_store = []  # list of dicts: {embedding, query, response}\n\ndef find_match(query_embedding, threshold=0.92):\n    best_score, best_entry = 0, None\n    for entry in cache_store:\n        score = float(np.dot(query_embedding, entry[\"embedding\"]))\n        if score > best_score:\n            best_score, best_entry = score, entry\n    if best_score > threshold:\n        return best_entry\n    return None\nphp\ndef semantic_cached_call(prompt: str) -> str:\n    query_embedding = embed(prompt)\n    match = find_match(query_embedding)\n\n    if match:\n        print(f\"cache hit (score matched threshold)\")\n        return match[\"response\"]\n\n    response = call_llm(prompt)  # your actual OpenAI/Anthropic call\n    cache_store.append({\n        \"embedding\": query_embedding,\n        \"query\": prompt,\n        \"response\": response,\n    })\n    return response\n```\n\nThat's the whole thing. Run it against a batch of paraphrased test queries and you'll immediately see cache hits on things a string cache would have completely missed.\n\nTry this:\n\n```\nprint(semantic_cached_call(\"How do I cancel my subscription?\"))\nprint(semantic_cached_call(\"How do I reactivate my subscription?\"))\n```\n\nDepending on your embedding model and threshold, there's a real chance the second query gets served the *cancel* answer, because \"cancel\" and \"reactivate\" share almost every other word in the sentence and embedding models cluster on shared vocabulary more than they distinguish directive verbs. This is the single most common failure mode in semantic caching, and it's why you can't ship the 40-line version above without a guard rail.\n\nHere's the fix a lightweight rejection rule for known polarity pairs, applied *after* the similarity match, before you trust it:\n\n```\nOPPOSING_PAIRS = [\n    (\"cancel\", \"reactivate\"), (\"enable\", \"disable\"),\n    (\"add\", \"remove\"), (\"increase\", \"decrease\"),\n]\n\ndef has_conflicting_intent(query: str, cached_query: str) -> bool:\n    q, c = query.lower(), cached_query.lower()\n    for a, b in OPPOSING_PAIRS:\n        if (a in q and b in c) or (b in q and a in c):\n            return True\n    return False\n\ndef safer_semantic_call(prompt: str) -> str:\n    query_embedding = embed(prompt)\n    match = find_match(query_embedding)\n\n    if match and not has_conflicting_intent(prompt, match[\"query\"]):\n        return match[\"response\"]\n\n    response = call_llm(prompt)\n    cache_store.append({\"embedding\": query_embedding, \"query\": prompt, \"response\": response})\n    return response\n```\n\nThis won't catch every edge case — for that you'd want a proper cross-encoder rerank step or a small intent classifier — but it eliminates the single most embarrassing class of bug for almost no added complexity, and it's a good stopping point for a weekend project or an internal tool.\n\nThe in-memory list works for a demo. For real traffic, swap it for an actual vector database the query pattern is the same, you're just trading `for entry in cache_store`\n\nfor an ANN index:\n\n``` python\n# Using Qdrant as an example\nfrom qdrant_client import QdrantClient\nfrom qdrant_client.models import PointStruct, VectorParams, Distance\n\nclient = QdrantClient(\":memory:\")  # swap for a real host in prod\nclient.create_collection(\n    collection_name=\"semantic_cache\",\n    vectors_config=VectorParams(size=384, distance=Distance.COSINE),\n)\n\ndef find_match_qdrant(query_embedding, threshold=0.92):\n    results = client.search(\n        collection_name=\"semantic_cache\",\n        query_vector=query_embedding.tolist(),\n        limit=1,\n    )\n    if results and results[0].score > threshold:\n        return results[0].payload\n    return None\n```\n\nSame logic, now backed by an index that scales to millions of cached entries with sub-millisecond lookup.\n\n**Mistake 1: One global threshold for every query type.** A threshold tuned for \"how do I use the API\" is too loose for \"what's my refund eligibility.\" If your app spans multiple domains, use different thresholds or exclude sensitive categories from caching entirely.\n\n**Mistake 2: Caching everything, forever.** Cached answers go stale the moment the underlying facts change (pricing, policies, feature availability). Add a `created_at`\n\nto every cache entry and expire aggressively for anything that isn't evergreen.\n\n**Mistake 3: Trusting similarity score alone.** As shown above, high similarity doesn't mean same intent. Always pair vector search with at least one lightweight guard rail before serving a cached response.\n\n**Mistake 4: Not measuring hit rate by category.** Overall hit rate is a vanity metric. A support bot might get 70% hits on \"how do I...\" questions and near 0% on account-specific ones know which is which so you can decide what's worth caching at all.\n\nTake the 40-line version above, hook it up to whatever LLM you're already calling, and log cache hits vs. misses for a day of real traffic. I'd genuinely bet you find at least a 30–40% hit rate on any app with repetitive user intents (support, FAQs, internal tooling) and every one of those hits is a full LLM call you just didn't have to pay for or wait on.\n\nIf you want a more complete reference implementation with the guard-rail and invalidation logic built in, I put one together as an open-source package called [Remem](https://github.com/hrsvd/remem) (`pip install remem-ai`\n\n) worth a look once you've built your own version and want to compare notes on the harder edge cases.\n\n**Your turn:** build the 40-line version, run it against your own logs or a paraphrased test set, and tell me your hit rate in the comments. I want to know if 30–40% holds up outside my own traffic patterns.", "url": "https://wpnews.pro/news/build-a-semantic-cache-for-your-llm-app-in-40-lines-of-python-and-cut-costs-by", "canonical_source": "https://dev.to/hrsvd/build-a-semantic-cache-for-your-llm-app-in-40-lines-of-python-and-cut-costs-by-half-2216", "published_at": "2026-08-03 08:21:34+00:00", "updated_at": "2026-08-03 08:44:29.957476+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "machine-learning"], "entities": ["Python", "Redis", "SentenceTransformer", "OpenAI", "Anthropic", "Qdrant", "Milvus"], "alternates": {"html": "https://wpnews.pro/news/build-a-semantic-cache-for-your-llm-app-in-40-lines-of-python-and-cut-costs-by", "markdown": "https://wpnews.pro/news/build-a-semantic-cache-for-your-llm-app-in-40-lines-of-python-and-cut-costs-by.md", "text": "https://wpnews.pro/news/build-a-semantic-cache-for-your-llm-app-in-40-lines-of-python-and-cut-costs-by.txt", "jsonld": "https://wpnews.pro/news/build-a-semantic-cache-for-your-llm-app-in-40-lines-of-python-and-cut-costs-by.jsonld"}}