# Semantic Cache in AI Tokenomics > Source: > Published: 2026-08-16 15:58:37+00:00 A few weeks ago I was looking at logs for a support bot that answers the same handful of questions all day long. "How do I reset my password", "how do I reset my login", "I forgot my password, help", "can't log in, need to reset it". Four different customers, four different sentences, one single question underneath all of them. The bot had a cache. It just never hit. Every single one of those four questions triggered a full model call, because the cache was checking for an exact string match, and none of those sentences matched each other character for character. That is the moment it clicked. Most teams think their caching problem is "we need a cache." The real problem is usually "we have a cache and it is nearly useless," because it only fires when someone happens to type the exact same words as someone before them. In practice that almost never happens. People ask the same question in a dozen different ways, and an exact match cache is blind to every single one of those variations except the one it has seen before, word for word. The fix is not a bigger cache. It is a different kind of cache, one that checks meaning instead of spelling. A normal cache works off a hash of the input. Same input, same hash, same cached response. It is fast, it is simple, and for a huge class of problems it works great. It falls apart the moment your input is natural language written by different humans, because natural language is never exactly the same twice. "What are your hours" and "when are you open" mean the same thing to a person and mean nothing alike to a hash function. So the cache stores the first one, and then completely misses the second, third, and fourth time someone asks the same question in slightly different words. You can watch this happen in your own logs. Pull up a week of prompts sent to any customer facing AI feature and group them by rough topic instead of exact text. You will almost always find clusters of near identical questions that never once hit each other in cache, because none of them were spelled exactly the same. A semantic cache does not ask "have I seen this exact text before." It asks "have I seen something that means roughly this before." It does that with three ingredients: Say your cache already has this one entry stored: | New question | Similarity to cached question | Above 0.92 threshold? | What happens | |---|---|---|---| | "How do I reset my login" | 0.96 | Yes | Cached answer is reused, no model call | | "I forgot my password, help" | 0.94 | Yes | Cached answer is reused, no model call | | "How do I delete my account" | 0.61 | No | Different question, model is called, new answer gets cached too | Notice the first two questions do not share many exact words with the cached one, "reset my login" versus "reset my password" barely overlaps as text. But they mean almost the same thing, so their embeddings land close together, and the cache correctly treats them as the same question. The third question is genuinely different, so its embedding lands far away, and it correctly triggers a real model call instead of returning the wrong cached answer. First, a small helper that measures how similar two embeddings are. This is the "how close are these two meanings" part: ``` python import numpy as np def similarity(embedding_a, embedding_b): """Returns a number between 0 and 1. Closer to 1 means the two pieces of text mean something very similar.""" return np.dot(embedding_a, embedding_b) / ( np.linalg.norm(embedding_a) * np.linalg.norm(embedding_b) ) ``` Next, the cache itself. It just keeps a list of everything it has answered before, and checks new questions against that list: ``` python class SemanticCache: def __init__(self, similarity_threshold=0.92): self.threshold = similarity_threshold self.stored_answers = [] # list of (embedding, original_question, answer) def find_cached_answer(self, new_embedding): """Looks for a past question that means the same thing as this one. Returns the old answer if found, otherwise returns None.""" best_answer = None best_similarity = 0 for old_embedding, old_question, old_answer in self.stored_answers: score = similarity(new_embedding, old_embedding) if score > best_similarity: best_similarity = score best_answer = old_answer if best_similarity >= self.threshold: return best_answer return None # nothing close enough, this is a genuinely new question def save_answer(self, embedding, question, answer): self.stored_answers.append((embedding, question, answer)) ``` Finally, here is how you would actually use it when a question comes in: ``` python def answer_question(question_text, cache, embed_fn, ask_model_fn): # Step 1: turn the new question into numbers embedding = embed_fn(question_text) # Step 2: check if we already answered something similar cached_answer = cache.find_cached_answer(embedding) if cached_answer is not None: return cached_answer # free, no model call needed # Step 3: no similar question found, so ask the model for real answer = ask_model_fn(question_text) # Step 4: remember this for next time cache.save_answer(embedding, question_text, answer) return answer ``` That is the whole mechanism. A question comes in, it gets turned into numbers, those numbers get compared against everything answered before, and only a genuinely new question ever reaches the model. The interesting engineering work is not in this core logic, it is in tuning the threshold and handling the edge cases below so this does not go wrong. The threshold is doing all the work. Set it too low and you will start returning cached answers for questions that only sound similar but actually need different answers, which is a much worse failure mode than a cache miss, because now you are confidently wrong instead of just slightly slower. Set it too high and you are back to almost never hitting the cache, which defeats the purpose. A few things that help in practice: **Scope the cache per intent, not globally.** Comparing every new question against every cached entry gets slow and increases the chance of a false match as your cache grows. Bucket entries by rough category first, like "billing questions" versus "technical support questions", and only compare within the matching bucket. **Treat a cache hit as a starting point for high stakes answers, not a final answer.** For low stakes questions like store hours, returning a cached answer straight away is fine. For anything involving account specific details, use the cache hit to decide "this looks like a duplicate question" and then still verify against current data before answering, rather than returning old cached content wholesale. **Expire entries.** A cached answer to "what is your current pricing" from three months ago is actively harmful once pricing changes. Attach a time to live to entries, and invalidate anything tied to information that changes on its own schedule. **Log near misses.** Track cases where similarity was close to your threshold but did not quite cross it. That is exactly the data you need to tune the threshold correctly instead of guessing at a number. The savings show up in exactly the places exact match caching misses. Any feature that fields the same handful of real questions phrased a dozen different ways, support bots, FAQ assistants, internal help desks, onboarding flows, all of these tend to have a small number of actual distinct questions hiding behind a much larger number of distinct phrasings. Semantic caching is what lets you collapse those phrasings back down to the handful of questions they actually represent, and pay for the model call once per question instead of once per phrasing. If you already have a cache and it still feels like your AI bill is too high, the cache is probably not the problem, the matching logic underneath it is. Pull a sample of your actual production prompts, group them by what they are really asking, and count how many of those groups your current cache is completely blind to. That number is usually bigger than people expect, and it is the clearest sign that an exact match cache has quietly stopped doing its job.