Python Semantic Cache: Cut Free-Tier LLM Token Costs

A developer built a Python semantic cache that cuts free-tier LLM token costs by matching new questions by meaning and returning stored answers when embedding distance is close enough. The cache stores past responses on disk, uses NumPy and JSON, and trades a cheap embedding call for an expensive chat completion on repetitive workloads. The implementation was prepared as part of MonkeyCode's product outreach.

A local semantic cache in Python cuts free-tier LLM token costs by matching new questions by meaning and returning a stored answer when embedding distance is close enough. I spend a cheap embedding call instead of another expensive chat completion whenever the intent is the same. I built this cache for repetitive workloads on free-tier APIs. It stores past responses on disk, runs with NumPy and JSON, and works with free model access. Every stage below has a verification step so you can prove the hit before you trust it. Look at real usage, not demo prompts. A support bot answers the same issues daily. A content pipeline summarizes similar documents. A test suite hits the same prompts weekly. Users almost never type the same sentence twice, so every paraphrase becomes another billed completion. Exact-match caches fail here. "How do I reset my password?" and "I forgot my password, help" are the same intent. A string hash sees two different inputs and charges you twice. Semantic caching converts text to embeddings and compares distances. Close enough means cached. Compared with exact match: The embedding call is cheap. The chat-model call is expensive. You trade a small cost for a large one, which is the whole point on a free token allowance. Three pieces are required: The request flow is: php New question - embed it - search stored vectors - closest match below threshold? - return cached response - no match? - call the model - store the response - return it I use Euclidean distance numpy.linalg.norm of the vector difference because it is one line of NumPy https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html and needs no extra library. If you later switch to cosine similarity, retune the threshold; do not reuse 0.15 blindly. The store can be a JSON file on disk. That is enough to learn the pattern before you introduce a vector database. Here is a minimal implementation. It uses NumPy for vector math and Python's json https://docs.python.org/3/library/json.html module for storage. No database required. python import json import numpy as np from pathlib import Path class SemanticCache: def init self, embed fn, threshold=0.15, path="cache.json" : self.embed fn = embed fn self.threshold = threshold self.path = Path path self.items = self. load def load self : if not self.path.exists : return return json.loads self.path.read text def save self : self.path.write text json.dumps self.items def get self, question: str : q vec = np.array self.embed fn question for item in self.items: stored vec = np.array item "embedding" dist = np.linalg.norm q vec - stored vec if dist < self.threshold: return item "response" return None def put self, question: str, response: str : vec = self.embed fn question self.items.append { "question": question, "embedding": vec.tolist , "response": response, } self. save The threshold is the key parameter. Too low means misses. Too high means wrong answers. I start at 0.15 for this L2 setup and tune from measured distances, not from a guess. You need an embedding model. Options vary by ecosystem. For this tutorial, the reference implementation uses an embedding endpoint available through MonkeyCode, an open-source project with free model access. The current allowance is 10 million tokens. Check the official docs for current terms. Disclosure: This article was prepared as part of MonkeyCode's product outreach. php def embed text: str - list float : resp = client.embeddings.create model="your-embedding-model", input=text, return resp.data 0 .embedding The embedding model does not need to match the chat model. Use the cheapest embedding model you can find. Embeddings are reusable across providers if you keep the same model. If you change models, rebuild the cache—vectors from different models are not comparable. For background on why dense vectors capture paraphrase, see OpenAI's embeddings guide https://platform.openai.com/docs/guides/embeddings as a conceptual reference; the same idea applies regardless of vendor. Combine the cache with your model client. The wrapper checks the cache first. php def cached complete question: str - str: cached = cache.get question if cached is not None: return cached response = model complete question cache.put question, response return response This is the entire integration. Call cached complete everywhere you previously called model complete . Verify: Run the same question twice. The first call hits the model. The second returns instantly. Check the cache file to confirm the entry exists. A cache without metrics is a guess. Add counters so a session ends with a number, not a feeling. python class CacheStats: def init self : self.hits = 0 self.misses = 0 def hit rate self : total = self.hits + self.misses return self.hits / total if total else 0 stats = CacheStats def cached complete question: str - str: cached = cache.get question if cached is not None: stats.hits += 1 return cached stats.misses += 1 response = model complete question cache.put question, response return response Log the hit rate after each session. A healthy cache for a support bot should exceed 40 percent. A content pipeline with unique inputs might sit near zero. If your hit rate is low, your workload is not repetitive. The cache is not helping, and you should not keep paying for embeddings that never save a completion. The threshold needs validation. Write a small test with known pairs before you put it in front of users. python def test threshold : pairs = "reset my password", "I forgot my password" , "refund policy", "what is your return policy" , "hello", "goodbye" , should NOT match for a, b in pairs: dist = np.linalg.norm np.array embed a - np.array embed b print f"{a} <- {b}: {dist:.3f}" Run this before deploying. Look at the distances. Set the threshold between the similar-pair distances and the dissimilar-pair distances. That gap is your safety margin. Verify: Similar pairs should sit below your threshold. The dissimilar pair should sit above it. If not, pick a different embedding model or adjust the threshold. The cache needs to run where your real traffic runs. MonkeyCode offers a free server option. You can deploy this cache there. Confirm the current plan before relying on it. Deploy the project. The cache file persists on disk. That is enough for a single instance. after deployment curl -X POST /complete -d '{"question": "how do I reset my password"}' curl -X POST /complete -d '{"question": "I forgot my password"}' The second request should return from cache. Measure the response time difference. The cached response should be dramatically faster. Keep these production limits in mind: Who should not use this: Those teams should skip caching entirely. Repetition is expensive. Semantic caching removes the repetition with a small Python class, an embedding function, and a threshold. It runs on a free server and stretches a free token allowance further. Measure your hit rate, tune your threshold, and watch your cache size. Then let the cache absorb the boring questions while the model handles the new ones. Try it today: wire the class to your embedding and chat clients, run the two verification steps same question twice, then the known-pair distance test , and log hit rate for one real session. Decide with that number—not with a hunch—whether the cache belongs in your path. If you ship it, share the hit rate you observed and the threshold you landed on so the next person does not start from zero.