{"slug": "kmemo-a-semantic-cache-for-llm-calls-that-refuses-to-serve-you-the-wrong-answer", "title": "Kmemo: a semantic cache for LLM calls that refuses to serve you the wrong answer", "summary": "Kmemo is a semantic cache for LLM calls that uses similarity as a first filter, then applies a chain of ten guards to reject near misses that would produce wrong answers, such as swapped numbers or mismatched units. The Java/Kotlin library requires JDK 17+ and ships with guard packs for English, Italian, Spanish, German, and French.", "body_md": "An exact-match cache misses \"how do I reverse a list in Python\" when it has already answered \"python list reverse\". A semantic cache doesn't: it embeds the prompt, finds the closest one it has seen, and replays that answer instead of calling the model. Fewer API calls, lower latency, same answers.\n\nExcept for the part where it hands back the wrong one.\n\n```\n\"Convert 100 USD to EUR\"\n\"Convert 250 USD to EUR\"      cosine similarity: ~0.99\n```\n\nEvery mainstream embedding model scores that pair around 0.99. No threshold separates it from a genuine paraphrase, because on the similarity axis the near miss sits closer than most paraphrases do. Raise the threshold and you lose real hits before you lose that one.\n\nSo a cache built on a threshold alone will tell someone that 250 dollars is 92 euros. Quickly, with no error, and nothing in the logs.\n\n[Kmemo](https://github.com/NaCode-Studios/Kmemo) treats that as the main event rather than an edge case. Similarity is only the first filter. Candidates that clear it get read as text by a chain of ten guards looking for concrete evidence that the two answers must differ: swapped numbers, mismatched units, different entities, different time references, negation, flipped antonyms, reversed comparisons, or a different kind of answer being asked for.\n\nThe defaults follow from the cost asymmetry. A wrong rejection costs one API call. A wrong acceptance costs a wrong answer. So the guards abstain rather than guess.\n\nRequires JDK 17+.\n\n```\ndependencies {\n    implementation(\"io.github.nacode-studios:kmemo-core:1.0.0\")\n}\n```\n\n`kmemo-core`\n\ndeclares `kotlinx-coroutines-core`\n\nas its only dependency. You bring the embedding source, which is any function from `String`\n\nto `FloatArray`\n\n. Kmemo ships none and depends on no provider SDK.\n\n``` php\nval cache = SemanticCache(\n    embedder = Embedder { text -> openAi.embed(text) },\n    store = InMemoryStore(maxEntries = 10_000, ttl = 1.hours),\n)\n\nval answer = cache.getOrPut(prompt) { llm.complete(it) }\n```\n\n`getOrPut`\n\nembeds the prompt once and reuses the vector for both the lookup and the write. Concurrent callers asking the same thing get coalesced: the first one computes, the rest wait and are served its answer.\n\nA cache with a 4% hit rate is untunable unless you know what caused the misses, because the fix is opposite for a threshold miss and a guard rejection:\n\n``` php\nwhen (val result = cache.lookup(prompt)) {\n    is CacheLookup.Hit  -> result.response\n    is CacheLookup.Miss -> when (result.reason) {\n        MissReason.BELOW_THRESHOLD   -> // traffic repeats less than you assumed, or the threshold is too tight\n        MissReason.REJECTED_BY_GUARD -> // a guard found a concrete difference; result.detail says which\n        else -> null\n    }\n}\n```\n\nThere's also `cache.explain(prompt)`\n\n, a read-only companion that shows every candidate with every guard's verdict. It's what you reach for when a hit you expected didn't happen.\n\nAnything that changes what a correct answer looks like belongs in the scope: model, temperature, system prompt, tenant, language. Leave it out and the cache will serve one model's answer to another model's caller.\n\n```\ncache.getOrPut(prompt, scope = \"gpt-4o|t=0.0|v3\") { llm.complete(it) }\nSemanticCache(embedder)                                    // MatchGuards.standard()\nSemanticCache(embedder, guards = MatchGuards.strict())     // trades hit rate for margin\nSemanticCache(embedder, guards = MatchGuards.none())       // the naive similarity-only baseline\n```\n\nThe guards work outside English too. Curated packs ship for Italian, Spanish, German and French, each measured against a localized near-miss corpus:\n\n```\nSemanticCache(embedder, guards = MatchGuards.standard(Locale.ITALIAN))\n```\n\nAbout a third of near misses need world knowledge. Deworming a puppy is not the same as deworming an adult dog. The boiling point of ethanol is not the boiling point of methanol. No amount of token comparison catches those.\n\nFor that there's an optional `Verifier`\n\n, typically a cheap model call. It runs only on candidates that already cleared the threshold and every guard, so it costs nothing in the common case, and it fails closed: a timeout or an error rejects rather than serving something unconfirmed.\n\nThe guards are judged against three labelled corpora with a blind validation split that no guard was tuned against, run as a CI regression gate on every build.\n\nOn the blind split, near misses are rejected 67% of the time and paraphrases are kept 88% of the time.\n\nNeither number is 100%, and I would rather publish them than a marketing claim. The near misses that get through are mostly the world-knowledge cases the verifier covers. Reproduce them yourself:\n\n```\n./gradlew :kmemo-core:test --tests '*CorpusTest*'\n```\n\nHow the blind splits grow without getting contaminated is written up in [docs/CORPUS.md](https://github.com/NaCode-Studios/Kmemo/blob/main/docs/CORPUS.md).\n\n`ThresholdCalibrator`\n\nmeasures the right threshold for your embedding model. The value you found in a blog post was tuned for somebody else's.\n\n`Embedder`\n\nand `CacheStore`\n\nare one-method seams, so you can start in memory and move to a vector database without touching the match logic. Redis (RediSearch KNN) and Postgres (pgvector) stores ship, plus an opt-in in-process HNSW store for when the exact scan stops scaling.\n\nThe embedder is a network call on every lookup, so Kmemo lets you own its failure:\n\n```\nval cache = SemanticCache(\n    embedder = myEmbedder.retrying(maxAttempts = 4),\n    embedFailurePolicy = EmbedFailurePolicy.FALL_BACK_TO_COMPUTE,\n    negativeCacheSize = 10_000,\n)\ncache.warm(faqPairs.map { WarmEntry(it.question, it.answer) })\n```\n\nFor dashboards and logs, subscribe to the event stream instead of polling `stats()`\n\n. It costs nothing when unused:\n\n```\nval metrics = KmemoMetrics().also { it.bindTo(meterRegistry) }   // kmemo-micrometer\nval cache = SemanticCache(embedder, listeners = listOf(metrics, Slf4jCacheListener()))\n```\n\n`SemanticCache`\n\nbean`Advisor`\n\nfor `ChatClient`\n\n`ChatModel`\n\nwrapper[ examples/](https://github.com/NaCode-Studios/Kmemo/tree/main/examples) is a runnable demo that needs no API key. It shows a guard catching a live near miss, with a\n\n`docker-compose`\n\nfor the Redis store.`io.github.nacode-studios:kmemo-core:1.0.0`\n\n`1.0`\n\nis stable under SemVer. If you have run a semantic cache in production and hit a false hit I haven't thought about, I want to hear about it. Open an issue with the pair that broke it.", "url": "https://wpnews.pro/news/kmemo-a-semantic-cache-for-llm-calls-that-refuses-to-serve-you-the-wrong-answer", "canonical_source": "https://dev.to/tonytonycoder11/kmemo-a-semantic-cache-for-llm-calls-that-refuses-to-serve-you-the-wrong-answer-54h7", "published_at": "2026-07-25 18:50:51+00:00", "updated_at": "2026-07-25 19:31:40.745306+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Kmemo", "NaCode Studios", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/kmemo-a-semantic-cache-for-llm-calls-that-refuses-to-serve-you-the-wrong-answer", "markdown": "https://wpnews.pro/news/kmemo-a-semantic-cache-for-llm-calls-that-refuses-to-serve-you-the-wrong-answer.md", "text": "https://wpnews.pro/news/kmemo-a-semantic-cache-for-llm-calls-that-refuses-to-serve-you-the-wrong-answer.txt", "jsonld": "https://wpnews.pro/news/kmemo-a-semantic-cache-for-llm-calls-that-refuses-to-serve-you-the-wrong-answer.jsonld"}}