# Kmemo: a semantic cache for LLM calls that refuses to serve you the wrong answer > Source: > Published: 2026-07-25 18:50:51+00:00 An exact-match cache misses "how do I reverse a list in Python" when it has already answered "python list reverse". A semantic cache doesn't: it embeds the prompt, finds the closest one it has seen, and replays that answer instead of calling the model. Fewer API calls, lower latency, same answers. Except for the part where it hands back the wrong one. ``` "Convert 100 USD to EUR" "Convert 250 USD to EUR" cosine similarity: ~0.99 ``` Every mainstream embedding model scores that pair around 0.99. No threshold separates it from a genuine paraphrase, because on the similarity axis the near miss sits closer than most paraphrases do. Raise the threshold and you lose real hits before you lose that one. So a cache built on a threshold alone will tell someone that 250 dollars is 92 euros. Quickly, with no error, and nothing in the logs. [Kmemo](https://github.com/NaCode-Studios/Kmemo) treats that as the main event rather than an edge case. Similarity is only the first filter. Candidates that clear it get read as text by a chain of ten guards looking for concrete evidence that the two answers must differ: swapped numbers, mismatched units, different entities, different time references, negation, flipped antonyms, reversed comparisons, or a different kind of answer being asked for. The defaults follow from the cost asymmetry. A wrong rejection costs one API call. A wrong acceptance costs a wrong answer. So the guards abstain rather than guess. Requires JDK 17+. ``` dependencies { implementation("io.github.nacode-studios:kmemo-core:1.0.0") } ``` `kmemo-core` declares `kotlinx-coroutines-core` as its only dependency. You bring the embedding source, which is any function from `String` to `FloatArray` . Kmemo ships none and depends on no provider SDK. ``` php val cache = SemanticCache( embedder = Embedder { text -> openAi.embed(text) }, store = InMemoryStore(maxEntries = 10_000, ttl = 1.hours), ) val answer = cache.getOrPut(prompt) { llm.complete(it) } ``` `getOrPut` embeds the prompt once and reuses the vector for both the lookup and the write. Concurrent callers asking the same thing get coalesced: the first one computes, the rest wait and are served its answer. A cache with a 4% hit rate is untunable unless you know what caused the misses, because the fix is opposite for a threshold miss and a guard rejection: ``` php when (val result = cache.lookup(prompt)) { is CacheLookup.Hit -> result.response is CacheLookup.Miss -> when (result.reason) { MissReason.BELOW_THRESHOLD -> // traffic repeats less than you assumed, or the threshold is too tight MissReason.REJECTED_BY_GUARD -> // a guard found a concrete difference; result.detail says which else -> null } } ``` There's also `cache.explain(prompt)` , a read-only companion that shows every candidate with every guard's verdict. It's what you reach for when a hit you expected didn't happen. Anything that changes what a correct answer looks like belongs in the scope: model, temperature, system prompt, tenant, language. Leave it out and the cache will serve one model's answer to another model's caller. ``` cache.getOrPut(prompt, scope = "gpt-4o|t=0.0|v3") { llm.complete(it) } SemanticCache(embedder) // MatchGuards.standard() SemanticCache(embedder, guards = MatchGuards.strict()) // trades hit rate for margin SemanticCache(embedder, guards = MatchGuards.none()) // the naive similarity-only baseline ``` The guards work outside English too. Curated packs ship for Italian, Spanish, German and French, each measured against a localized near-miss corpus: ``` SemanticCache(embedder, guards = MatchGuards.standard(Locale.ITALIAN)) ``` About a third of near misses need world knowledge. Deworming a puppy is not the same as deworming an adult dog. The boiling point of ethanol is not the boiling point of methanol. No amount of token comparison catches those. For that there's an optional `Verifier` , typically a cheap model call. It runs only on candidates that already cleared the threshold and every guard, so it costs nothing in the common case, and it fails closed: a timeout or an error rejects rather than serving something unconfirmed. The guards are judged against three labelled corpora with a blind validation split that no guard was tuned against, run as a CI regression gate on every build. On the blind split, near misses are rejected 67% of the time and paraphrases are kept 88% of the time. Neither number is 100%, and I would rather publish them than a marketing claim. The near misses that get through are mostly the world-knowledge cases the verifier covers. Reproduce them yourself: ``` ./gradlew :kmemo-core:test --tests '*CorpusTest*' ``` How the blind splits grow without getting contaminated is written up in [docs/CORPUS.md](https://github.com/NaCode-Studios/Kmemo/blob/main/docs/CORPUS.md). `ThresholdCalibrator` measures the right threshold for your embedding model. The value you found in a blog post was tuned for somebody else's. `Embedder` and `CacheStore` are one-method seams, so you can start in memory and move to a vector database without touching the match logic. Redis (RediSearch KNN) and Postgres (pgvector) stores ship, plus an opt-in in-process HNSW store for when the exact scan stops scaling. The embedder is a network call on every lookup, so Kmemo lets you own its failure: ``` val cache = SemanticCache( embedder = myEmbedder.retrying(maxAttempts = 4), embedFailurePolicy = EmbedFailurePolicy.FALL_BACK_TO_COMPUTE, negativeCacheSize = 10_000, ) cache.warm(faqPairs.map { WarmEntry(it.question, it.answer) }) ``` For dashboards and logs, subscribe to the event stream instead of polling `stats()` . It costs nothing when unused: ``` val metrics = KmemoMetrics().also { it.bindTo(meterRegistry) } // kmemo-micrometer val cache = SemanticCache(embedder, listeners = listOf(metrics, Slf4jCacheListener())) ``` `SemanticCache` bean`Advisor` for `ChatClient` `ChatModel` wrapper[ examples/](https://github.com/NaCode-Studios/Kmemo/tree/main/examples) is a runnable demo that needs no API key. It shows a guard catching a live near miss, with a `docker-compose` for the Redis store.`io.github.nacode-studios:kmemo-core:1.0.0` `1.0` is stable under SemVer. If you have run a semantic cache in production and hit a false hit I haven't thought about, I want to hear about it. Open an issue with the pair that broke it.