The Semantic Cache That Made a Free LLM Quota Feel Infinite

MonkeyCode has documented a zero-dependency semantic cache that can cut token consumption by roughly half in typical agent loops, addressing the inefficiency of repeated prompts in free-tier LLM quotas. The implementation uses character n-gram Jaccard similarity and a threshold of 0.92 to serve cached responses for semantically similar requests, reducing latency and cost without degrading answer quality for novel prompts.

A token allowance is usually treated as a spending budget, which is the wrong mental model for free tiers. The right model is a cache to be managed, because agent workloads repeat themselves far more than developers realize. A semantic cache that serves previous responses for rephrased requests can cut token consumption by roughly half in typical agent loops. This article documents a working zero-dependency implementation and the threshold tuning that made it safe enough for production prototypes. MonkeyCode's current free offering includes a 10-million-token allowance and a free server option, which makes quota efficiency a practical concern rather than a theoretical one. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The percentages and latency figures below are illustrative measurements from a controlled prototype, not guarantees of any specific result; the design pattern is the deliverable. Most developers assume that an agent loop sends mostly unique prompts, so they never measure how much repetition actually occurs. In a typical tool-calling loop, the same intent arrives in many surface forms: a user asks the same question twice with different wording, a retry resends a failed request with slightly modified phrasing, or multiple users query the same documentation. A quick audit of request logs in the prototype showed that roughly 35% of prompts were semantically near-duplicates of something already answered. Each one of those duplicate requests burns tokens, adds latency, and risks inconsistent answers when the model rephrases a response differently on the second pass. The fix is not to make the model smarter but to stop sending the duplicate at all. A semantic cache sits in front of the endpoint, computes a similarity score between the incoming prompt and previously cached prompts, and returns the stored response when the score clears a threshold. The cache does not replace the model; it reduces the number of calls that reach the model. That distinction matters, because a cache miss still goes to the endpoint with full fidelity, and the cache learns from every miss. The result is a system that gets cheaper and faster the longer it runs, without any degradation in answer quality for genuinely novel requests. The core design decision is how to measure whether two prompts mean the same thing. Production systems often use embedding models for this task, but that adds a dependency and its own API cost. A zero-dependency alternative is character n-gram Jaccard similarity, which tokenizes text into overlapping three-character shingles and compares the resulting sets. It handles paraphrases reasonably well for short prompts and requires no external services. The threshold controls the trade-off between hit rate and correctness. A threshold of 0.85 catches more duplicates but risks returning a cached answer for a prompt that differs in a semantically important way. A threshold of 0.95 is safer but misses many legitimate paraphrases. The prototype settled on 0.92 after running a calibration pass over one hundred logged prompts, which is a simple exercise: compute pairwise similarities, manually label which pairs are safe to deduplicate, and pick the threshold that separates the two clusters. The similarity metric choice depends on the prompt shape. Short tool-call instructions benefit from character n-grams because word-level overlap is too coarse. Longer prose prompts might justify a word-based Jaccard or a proper embedding model if the cache hit rate justifies the extra dependency. The implementation below uses character trigrams and works well for prompts under two hundred characters. A cache without eviction is a memory leak, so the storage layer needs a time-to-live policy. The prototype used SQLite with a created at timestamp and a TTL of one hour, which balances freshness against hit rate. A shorter TTL protects against stale answers but discards useful entries; a longer TTL improves hit rate but risks serving outdated information for time-sensitive queries. The schema stores the original prompt, the response text, the token cost of the original call, the creation time, and a hit counter. The hit counter enables a simple popularity-aware eviction strategy: when the table grows beyond a size limit, the query deletes the oldest entries with the lowest hit counts. This keeps the cache focused on the prompts that actually recur rather than one-off requests that will never repeat. The implementation below shows the full cache class with initialization, tokenization, similarity scoring, retrieval, and insertion. It is deliberately dependency-free so it runs on any fresh Python environment, including a free server with no package installation allowed. python import hashlib import sqlite3 import time class SemanticCache: def init self, db path="semantic cache.db", threshold=0.92, ttl seconds=3600 : self.conn = sqlite3.connect db path self.threshold = threshold self.ttl = ttl seconds self. init db def init db self : self.conn.execute """ CREATE TABLE IF NOT EXISTS cache prompt TEXT PRIMARY KEY, response TEXT, tokens INTEGER, created at REAL, hit count INTEGER DEFAULT 1 """ self.conn.commit def tokenize self, text : text = text.lower .strip if len text <= 3: return {text} return {text i:i+3 for i in range len text - 2 } def similarity self, a, b : if not a or not b: return 0.0 return len a & b / len a | b def get self, prompt : now = time.time rows = self.conn.execute "SELECT prompt, response, created at FROM cache WHERE created at ? ORDER BY hit count DESC LIMIT 100", now - self.ttl, .fetchall prompt grams = self. tokenize prompt best match = None best score = 0.0 for stored prompt, response, in rows: score = self. similarity prompt grams, self. tokenize stored prompt if score best score: best score = score best match = stored prompt, response if best match and best score = self.threshold: self.conn.execute "UPDATE cache SET hit count = hit count + 1 WHERE prompt = ?", best match 0 , self.conn.commit return best match 1 , best score return None, best score def put self, prompt, response, tokens : self.conn.execute "INSERT OR REPLACE INTO cache VALUES ?, ?, ?, ?, 1 ", prompt, response, tokens, time.time self.conn.commit The cache wraps the endpoint call in a cache-aside pattern: check the cache first, return the stored response on a hit, and on a miss call the endpoint and store the result. The wrapper below shows how to integrate the cache with any OpenAI-compatible completion endpoint using only the standard library. python import json import urllib.request def cached completion cache, endpoint, api key, messages, max tokens=256 : prompt = messages -1 "content" if messages else "" cached, score = cache.get prompt if cached: return {"cached": True, "similarity": round score, 3 , "text": cached} body = json.dumps { "model": "default", "messages": messages, "max tokens": max tokens, } .encode request = urllib.request.Request endpoint, data=body, headers={ "Authorization": f"Bearer {api key}", "Content-Type": "application/json", } with urllib.request.urlopen request, timeout=30 as response: payload = json.load response text = payload "choices" 0 .get "text", "" .strip tokens = payload.get "usage", {} .get "total tokens", 0 cache.put prompt, text, tokens return {"cached": False, "text": text} The wrapper returns a structured result that distinguishes cache hits from misses, which makes the hit rate measurable. It also records the token cost of the original call so the cache can report how many tokens it saved. This measurement loop is what turns the cache from a guess into an engineering decision. The integration point matters. The cache belongs at the highest level of the request path, not inside the model client, because the goal is to skip the network call entirely. Putting the cache inside the client still saves tokens but leaves the latency cost of a local function call, which is negligible compared to a network round trip. A cache that never hits is dead code, and a cache that hits too often may be returning stale answers. The prototype tracked two metrics: the hit rate over a sliding window and the drift between cached and fresh responses. The hit rate is straightforward to compute from the wrapper's return values; the drift requires a periodic audit where a sample of cached prompts is re-sent to the endpoint and the responses are compared. In the prototype run, the hit rate stabilized around 30% after the first hour of traffic, which translated to a 45% reduction in token consumption because cached responses were typically longer than the prompts that triggered them. The drift audit found no substantive differences in a sample of fifty re-sent prompts, largely because the TTL kept entries fresh and the workload was dominated by stable documentation queries. The audit frequency depends on how quickly the underlying data changes. A support bot answering questions about a stable product can audit daily. A news summarization agent should audit hourly or skip the cache entirely for time-sensitive categories. The decision table below summarizes which workloads benefit from semantic caching and which do not. | Workload pattern | Cache effectiveness | Reason | |---|---|---| | FAQ and documentation queries | High | Same intent recurs with different wording | | Tool-call extraction | Medium | Similar structure but parameter values change | | Code generation | Low | Context variations change the correct output | | Time-sensitive news summaries | Avoid | Stale answers are worse than no answer | | Multi-turn conversation | Medium | Cache only the first turn, not the full context | The most dangerous failure mode is a false positive: two prompts that look similar but require different answers. A user asking "how do I deploy to production" and "how do I deploy to production with zero downtime" share most of their trigrams, but the second prompt needs a different response. The 0.92 threshold catches many of these cases, but not all, which is why the TTL and the drift audit exist as guardrails. The second failure mode is context poisoning from cached responses that contain stale information. A response generated before a product change may remain in the cache for the full TTL, and every hit during that window propagates the outdated answer. Shortening the TTL reduces the risk but also reduces the hit rate, so the right value depends on how fast the underlying domain changes. The third failure mode is cache poisoning from a bad response. If the endpoint returns an error message or a hallucinated answer, the cache stores it and serves it to every similar prompt until the TTL expires. The mitigation is to validate responses before caching, which can be as simple as checking that the response is non-empty and contains no error markers, or as complex as a separate quality classifier for high-stakes domains. A semantic cache is the highest-leverage optimization available for free-tier LLM usage because it reduces both token consumption and latency without changing the model. The implementation shown here runs on any free server, requires no external dependencies, and pays for itself within the first hour of traffic. The threshold tuning and the drift audit are the price of safety, and they are cheap compared to the cost of duplicate requests. Teams that are prototyping agents, building internal tools, or running evaluation loops on a budget should treat a semantic cache as a standard component rather than an optional enhancement. The same design works against any OpenAI-compatible endpoint, including MonkeyCode's free model access and free server option, which makes the comparison cheap to run. A cached token is a token that never needs to be spent, and on a free tier that distinction is the difference between a project that runs out of runway and one that keeps going.