A local semantic cache in Python cuts free-tier LLM token costs by matching new questions by meaning and returning a stored answer when embedding distance is close enough. I spend a cheap embedding call instead of another expensive chat completion whenever the intent is the same.
I built this cache for repetitive workloads on free-tier APIs. It stores past responses on disk, runs with NumPy and JSON, and works with free model access. Every stage below has a verification step so you can prove the hit before you trust it.
Look at real usage, not demo prompts. A support bot answers the same issues daily. A content pipeline summarizes similar documents. A test suite hits the same prompts weekly. Users almost never type the same sentence twice, so every paraphrase becomes another billed completion.
Exact-match caches fail here. "How do I reset my password?" and "I forgot my password, help" are the same intent. A string hash sees two different inputs and charges you twice.
Semantic caching converts text to embeddings and compares distances. Close enough means cached. Compared with exact match:
The embedding call is cheap. The chat-model call is expensive. You trade a small cost for a large one, which is the whole point on a free token allowance.
Three pieces are required:
The request flow is:
New question
-> embed it
-> search stored vectors
-> closest match below threshold?
-> return cached response
-> no match?
-> call the model
-> store the response
-> return it
I use Euclidean distance (numpy.linalg.norm
of the vector difference) because it is one line of NumPy and needs no extra library. If you later switch to cosine similarity, retune the threshold; do not reuse 0.15
blindly. The store can be a JSON file on disk. That is enough to learn the pattern before you introduce a vector database.
Here is a minimal implementation. It uses NumPy for vector math and Python's json module for storage. No database required.
import json
import numpy as np
from pathlib import Path
class SemanticCache:
def __init__(self, embed_fn, threshold=0.15, path="cache.json"):
self.embed_fn = embed_fn
self.threshold = threshold
self.path = Path(path)
self.items = self._load()
def _load(self):
if not self.path.exists():
return []
return json.loads(self.path.read_text())
def _save(self):
self.path.write_text(json.dumps(self.items))
def get(self, question: str):
q_vec = np.array(self.embed_fn(question))
for item in self.items:
stored_vec = np.array(item["embedding"])
dist = np.linalg.norm(q_vec - stored_vec)
if dist < self.threshold:
return item["response"]
return None
def put(self, question: str, response: str):
vec = self.embed_fn(question)
self.items.append({
"question": question,
"embedding": vec.tolist(),
"response": response,
})
self._save()
The threshold is the key parameter. Too low means misses. Too high means wrong answers. I start at 0.15
for this L2 setup and tune from measured distances, not from a guess.
You need an embedding model. Options vary by ecosystem. For this tutorial, the reference implementation uses an embedding endpoint available through MonkeyCode, an open-source project with free model access. The current allowance is 10 million tokens. Check the official docs for current terms.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
def embed(text: str) -> list[float]:
resp = client.embeddings.create(
model="your-embedding-model",
input=text,
)
return resp.data[0].embedding
The embedding model does not need to match the chat model. Use the cheapest embedding model you can find. Embeddings are reusable across providers if you keep the same model. If you change models, rebuild the cache—vectors from different models are not comparable. For background on why dense vectors capture paraphrase, see OpenAI's embeddings guide as a conceptual reference; the same idea applies regardless of vendor.
Combine the cache with your model client. The wrapper checks the cache first.
def cached_complete(question: str) -> str:
cached = cache.get(question)
if cached is not None:
return cached
response = model_complete(question)
cache.put(question, response)
return response
This is the entire integration. Call cached_complete
everywhere you previously called model_complete
.
Verify: Run the same question twice. The first call hits the model. The second returns instantly. Check the cache file to confirm the entry exists.
A cache without metrics is a guess. Add counters so a session ends with a number, not a feeling.
class CacheStats:
def __init__(self):
self.hits = 0
self.misses = 0
def hit_rate(self):
total = self.hits + self.misses
return self.hits / total if total else 0
stats = CacheStats()
def cached_complete(question: str) -> str:
cached = cache.get(question)
if cached is not None:
stats.hits += 1
return cached
stats.misses += 1
response = model_complete(question)
cache.put(question, response)
return response
Log the hit rate after each session. A healthy cache for a support bot should exceed 40 percent. A content pipeline with unique inputs might sit near zero. If your hit rate is low, your workload is not repetitive. The cache is not helping, and you should not keep paying for embeddings that never save a completion.
The threshold needs validation. Write a small test with known pairs before you put it in front of users.
def test_threshold():
pairs = [
("reset my password", "I forgot my password"),
("refund policy", "what is your return policy"),
("hello", "goodbye"), # should NOT match
]
for a, b in pairs:
dist = np.linalg.norm(
np.array(embed(a)) - np.array(embed(b))
)
print(f"{a} <-> {b}: {dist:.3f}")
Run this before deploying. Look at the distances. Set the threshold between the similar-pair distances and the dissimilar-pair distances. That gap is your safety margin.
Verify: Similar pairs should sit below your threshold. The dissimilar pair should sit above it. If not, pick a different embedding model or adjust the threshold.
The cache needs to run where your real traffic runs. MonkeyCode offers a free server option. You can deploy this cache there. Confirm the current plan before relying on it.
Deploy the project. The cache file persists on disk. That is enough for a single instance.
curl -X POST /complete -d '{"question": "how do I reset my password"}'
curl -X POST /complete -d '{"question": "I forgot my password"}'
The second request should return from cache. Measure the response time difference. The cached response should be dramatically faster.
Keep these production limits in mind:
Who should not use this:
Those teams should skip caching entirely.
Repetition is expensive. Semantic caching removes the repetition with a small Python class, an embedding function, and a threshold. It runs on a free server and stretches a free token allowance further. Measure your hit rate, tune your threshold, and watch your cache size. Then let the cache absorb the boring questions while the model handles the new ones.
Try it today: wire the class to your embedding and chat clients, run the two verification steps (same question twice, then the known-pair distance test), and log hit rate for one real session. Decide with that number—not with a hunch—whether the cache belongs in your path. If you ship it, share the hit rate you observed and the threshold you landed on so the next person does not start from zero.