TL;DR
Traditional caching checks if you've seen the exact same input before. Semantic caching checks if you've seen a meaningfully similar input before — using embeddings instead of string matching. For LLM-backed apps, this can cut API costs by 30–70% and slash latency from seconds to milliseconds, without touching your prompts or model choice.
The Problem
If you're building anything on top of an LLM API, you've probably noticed: Users ask the same question in a dozen different ways ("What's your refund policy?" vs "How do refunds work?" vs "Can I get my money back?")
Standard caching (Redis, exact-key lookups) misses every one of these because the strings don't match
Every one of those near-duplicate questions triggers a full, billable model call
You're paying full price — and full latency — for redundant work.
What Semantic Caching Does Differently
Instead of hashing the raw input string, a semantic cache:
Embeds the incoming query into a vector
Searches a vector store for a "close enough" previous query (cosine similarity above some threshold, e.g. 0.92+)
If found, returns the cached response instantly
If not, calls the LLM, then stores the new query + response pair for next time
This turns your cache hit rate from "only literal repeats" into "anything the model would have answered the same way.
In production, swap the in-memory list for a vector database (Redis with vector search, Pinecone, Qdrant, or pgvector) so the cache survives restarts and scales past a few thousand entries.
Where It Shines
Customer support bots — huge overlap in phrasing across users
RAG systems — repeated questions against the same knowledge base
Internal dev tools — the same handful of "how do I..." queries over and over
High-traffic apps — where even a 20% cache hit rate meaningfully moves your bill
Where to Be Careful
Threshold tuning matters. Too loose, and you'll serve stale or wrong answers to subtly different questions ("cancel my subscription" vs "cancel my free trial" can embed close together but mean very different things).
Time-sensitive queries ("what's the weather," "latest price") shouldn't be cached at all — add a bypass list or intent classifier in front of the cache.
Cache invalidation is still the hard problem it's always been. If your underlying data changes, stale cached answers become a liability, not a feature.
Embedding cost isn't free — but it's typically 10-50x cheaper than a full completion call, so the math still favors caching for most workloads.
Bottom Line
Semantic caching isn't a new invention so much as an obvious idea whose time has come now that embeddings are cheap and fast. If your LLM costs are creeping up and your queries have any repetition in intent (even if not in wording), this is one of the highest-leverage, lowest-effort optimizations you can add this week.
Have you implemented semantic caching in production? What threshold and vector store worked for you? Drop it in the comments.