Semantic caching is the cheapest model call: the one you never make. After embeddings and vector databases became default RAG plumbing, teams reused the same vectors to match paraphrased user questions to prior answers.
Quick answers #
| Question | Direct answer |
|---|---|
| First tier? | Exact hash (SHA-256 of normalized prompt + model + tool schema version) |
| Second tier? | Embedding nearest neighbor with similarity threshold |
| Stacks with? | Provider prompt caching andKV cache economics |
| Biggest risk? | False-positive reuse when two questions look alike but need different actions |
| Measure? | Hit rate, $ saved, rollback count, human audit sample |
Two-tier architecture #
Production systems almost always stack tiers:
- Exact hash hit — zero embedding cost, safest reuse.
- Semantic near hit — embed the prompt, search a vector index of prior prompts, return the paired answer if similarity clears threshold.
- Miss — call the model, store result asynchronously.
The GPTCache documentation describes this pattern explicitly: cache manager, embedding function, similarity evaluator and storage adapter. The ideas are older than any one library; what changed in 2026 is that embedding APIs are cheap enough to run on every request when tier one misses.
A long gray road with a short green fast lane beside it, showing how semantic hits skip full model decode
Use the lab to feel how tightening the threshold trades cost for risk.
Normalization before you hash #
Cache keys must include everything that changes behavior:
- Model name and pinned version (or router slug)
- System prompt hash
- Tool definition schema version (tool definition design )
- Temperature and structured-output schema flags
- Locale and policy pack version
Strip volatile whitespace for hashing, but do not strip numbers if digits matter ("refund order 8842" is not "refund order 8843"). For agents, include the last user message only when the cached segment is single-turn classification; multi-turn caches need thread IDs and compaction boundaries (context engineering).
Embedding search details #
Choose an embedding model aligned with your RAG pipeline. Mismatched embedders between retrieval and cache search create blind spots.
Operational tips:
- Store prompt embedding ,answer text ,metadata (latency, model, timestamp, user segment).
- Use separate namespaces per tenant in multi-tenant SaaS.
- Apply TTL for policies that change (shipping dates, pricing).
- Keep a negative cache of prompts that must never hit (legal holds, abuse flags).
Similarity metric is usually cosine on L2-normalized vectors. Reindex when you rotate embedders; old vectors are not comparable.
Green pebbles clustered beside distant cream ones, showing embedding neighborhoods used in semantic LLM cache search
When semantic caching pays off #
| Workload | Typical win | Caveat |
|---|---|---|
| FAQ and doc bots | High hit rate on paraphrases | Update cache on doc publish |
| Intent routing | Cheap embed + tiny classifier | Wrong route is expensive |
| Summaries of static pages | Strong | Page edits invalidate |
| Coding agents | Low | Changing repo state breaks keys |
| Tool-heavy agents | Low unless step is read-only | Never cache side effects |
Our LightSociety simulation write-up noted FAISS semantic caches beside SHA exact caches at scale — the pattern is mainstream in research stacks, not only demos.
Safety and compliance #
Semantic hits are stochastic. Mitigations:
- Human review queue for similarities between configured warn band and accept threshold.
- Canary prompts injected hourly to detect drift.
- Per-segment policies — marketing chat can be loose; HIPAA workflows exact-hash only.
- Audit logs with prompt hash, neighbor ID, similarity score, and operator override flag.
For agent security, pair caching with runtime guards. AgentBeam, from the explainx.ai team, stops agents before dangerous actions — caching should never bypass those checks on tool paths.
Rollout plan #
- Shadow mode — compute what would have hit, but still call the model; log savings estimate.
- Exact tier only in production one week; measure baseline.
- Enable semantic tier at conservative threshold; sample 1 in 100 hits for manual review.
- Tune threshold down only with measured false-positive rate under your SLO.
- Wire cache busting to CMS webhooks andvector DB index rebuild jobs.
Instrument beside OpenRouter routing: a bad cache hit looks like model quality regression.
Storage options #
| Store | Fits when |
|---|---|
| Redis + vector module | Low latency, modest corpus |
| pgvector | You already run Postgres for app data |
| Qdrant / Weaviate | Dedicated vector ops team |
| In-process FAISS | Single-node prototypes only |
Persist answers encrypted at rest if prompts contain PII. Cache entries are as sensitive as chat logs.
Latency and cost math (worked example) #
Suppose a support bot answers 40,000 identical-policy questions per day at 1,200 output tokens each on a mid-tier API.
- No cache: 40k × full prefill + decode bill.
- Exact hash only: maybe 12% hits if users paste the same wording — modest savings.
- Semantic tier at conservative threshold: FAQ paraphrases often cluster; teams report incremental 25–45% hit rates once normalization includes model ID. Treat vendor case studies as upper bounds; run shadow mode.
Each hit avoids decode tokens entirely, which matters more than shaving prefill when answers are long. Pair with provider prompt caching so near-misses that still call the model pay less on repeated system prefixes.
Two dials tuned toward cost and power, showing the tradeoff when tuning cache hit rate versus answer risk
Add embedding cost: one small embed call per miss-tier lookup. If your embedder runs at 2 ms and $0.02 per million tokens, that is usually noise compared with a 1,200-token generation — unless you query a gigantic cache index without filtering. Partition indexes by product line.
Key design and invalidation #
Version every cache entry:
cache_key = sha256(
model_id + tool_schema_hash + norm(prompt) + locale + policy_pack
)
On deploy:
- Bump
tool_schema_hashwhenMCP tools change. - Purge namespaces tied to updated RAG corpora .
- Keep previous namespace read-only for seven days to debug regressions.
For multilingual apps, either embed in the user's language with a multilingual model (EmbeddingGemma) or disable semantic tier across languages until you have labeled data.
Observability fields #
Export these metrics to the same dashboard as model latency:
| Metric | Why |
|---|---|
cache_tier (exact / semantic / miss) |
Split savings |
similarity_score on semantic hits |
Threshold tuning |
neighbor_age_hours |
Stale answer risk |
rollback_count |
User clicked "wrong answer" |
tokens_avoided |
Finance reconciliation |
Alert when semantic hit rate jumps overnight — that often means a too-loose threshold or a poisoned neighbor cluster.
Agent and harness specifics #
Claude Code and Codex sessions change repository state every turn. Do not semantic-cache "fix this bug" prompts unless the working tree hash is in the key. Safe segments: linter rule classification, commit message format suggestions on identical diffs, static style guides.
For decision models, cache routing decisions with labeled audit trails; a wrong route is cheaper than a wrong generation but still costly.
Regulatory and retention #
Cached answers are stored chat completions. Apply the same retention, export, and deletion workflows you use for message logs. EU users requesting erasure need neighbor entries removed, not only primary logs.
Document in your privacy policy that paraphrased questions may retrieve prior answers stored in the cache layer. Semantic hits are not "new" model inference from the regulator's point of view if personal data is replayed verbatim.
Integration sketch (pseudo-flow) #
- HTTP handler receives user prompt.
- Normalize and hash; query exact store (Redis string or SQL row).
- On miss, embed prompt; top-k search vector store with metadata filters (tenant, product).
- If best similarity above threshold, return cached answer +
X-Cache: semanticheader. - Else invoke model; on success, async write exact and semantic entries.
- Sample logs to labeling queue.
Open-source references: GPTCache documents adapter interfaces; align your implementation with one adapter rather than forking concepts.
Common mistakes #
- One global threshold across products with different risk.
- Caching JSON tool calls without including arguments in the key.
- Ignoring multilingual paraphrases — embed in the user's language or use multilingual embedders (EmbeddingGemma patterns ).
- Skipping version pins when the model vendor silently updates weights.
Pairing with rerankers and RAG #
Semantic caching is not a substitute for retrieval. The correct order in many APIs:
- Retrieve chunks with vector search .
- Optionally rerank .
- Build the prompt with citations.
- Check answer cache for near-duplicate questions without changing retrieved chunks.
If retrieved chunks change but the user question repeats, invalidate semantic neighbors tied to old corpus versions. Store corpus_version in cache metadata.
Cold start and warmup #
New deploys have empty caches — expect full model cost until traffic repeats. Pre-warm with canonical questions from support macros. Avoid poisoning the cache with test harness gibberish; use a dedicated env=staging namespace.
When not to build it #
Teams under ~10k requests per day often win more from prompt caching and smaller models than from operating a vector cache. Build semantic caching when repetition is obvious in logs and embedding search cost is dwarfed by generation.
FAQ for operators #
Does semantic caching break evals? Run eval suites with cache disabled weekly. Compare cached production traffic separately.
Does it interact with speculative decoding? Server-side decode optimizations do not replace application-level answer reuse.
Local stacks? MLX guide covers prompt caching files; add your own SQLite + embedder for semantic tier on laptop demos.
Related reading #
Article accurate as of November 9, 2026. Revisit thresholds when you change models, tools or compliance scope.