The cheapest LLM request is the one you don't send. If the same question shows up twice, there's no reason to pay twice — the model's answer hasn't changed, and the user doesn't care where it came from.
That's all prompt caching is. You store the response the first time, and serve it from memory the next time the same request comes in. Done well, it takes 30–99% off the bill and knocks latency down to sub-millisecond. Done badly, it serves stale or wrong answers.
This post covers both: what caching is, the two kinds you'll run into, where each works, and how to enable it on LLM Gateway without touching your code.
A cache sits between your app and the LLM provider. When a request comes in:
The second request for the same question is a Redis lookup. It costs nothing. It returns in under a millisecond.
Without caching: With caching (second hit):
───────────────── ─────────────────────────
App → Provider App → Gateway cache
~800ms, $0.01 <1ms, $0.00
They both save money. They work differently and stack on each other.
Hash the full request, store the full response. Only hits when the request is identical: same model, same messages, same temperature, same tools.
Cache the prefix of a prompt — the system message, tool definitions, and any shared context — and reuse it across requests with different user messages at the end.
They're complementary. LLM Gateway's exact-match caching catches repeat queries at the edge; provider prefix caching reduces cost on everything else. You don't have to choose.
Say you're running a customer support bot. 50,000 requests/day. Average 2,000 input tokens (system prompt + context), 500 output tokens. On GPT-4o that's:
Per request: (2000/1M)*$2.50 + (500/1M)*$10 = $0.005 + $0.005 = $0.010
Daily: $500
Monthly: $15,000
Add exact-match caching with a modest 40% hit rate (support questions repeat more than you'd think):
Cache hits: 20,000 × $0.000 = $0
Cache misses: 30,000 × $0.010 = $300
Daily: $300 (40% saved)
Monthly: $9,000 (saving $6,000/month)
Now layer provider prefix caching on the remaining misses — the 2,000-token system prompt is identical across all of them, so ~80% of input tokens get the cached rate:
Effective input cost per miss: roughly halves
Cache misses: 30,000 × ~$0.0075 = $225
Daily: $225 (55% saved vs. baseline)
Monthly: $6,750 (saving $8,250/month)
The numbers scale with volume. The cache infrastructure costs the same whether you do 100 requests or 100 million.
Caching is not free lunch. Don't enable it blindly:
Rule of thumb: if your prompt sets temperature: 0
or the task is factual/deterministic, cache it. Otherwise, don't.
No code changes required. Three steps in the dashboard:
Requests just work. Cached responses show cost: 0
in the usage dashboard so you can measure your hit rate directly.
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://api.llmgateway.io/v1",
apiKey: process.env.LLM_GATEWAY_API_KEY,
});
// No special parameters needed — caching is automatic
// when enabled at the project level.
const response = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Summarize: ..." }],
temperature: 0,
});
Full docs: docs.llmgateway.io/features/caching.
The difference between 10% and 70% hit rates is usually prompt hygiene, not the cache itself.
temperature: 0
on deterministic work
Classification, extraction, routing, yes/no decisions — none of these benefit from sampling variation. temperature: 0
maximizes cache hits and produces more reliable outputs anyway.
// Bad: each of these is a unique cache key
"what are your hours?";
"What are your hours?";
"what are your hours? ";
"What are your hours";
// Good: normalize once, hit the cache every time
const normalized = input
.trim()
.toLowerCase()
.replace(/[?.!]+$/, "");
Lowercase, trim whitespace, collapse punctuation. Small change, big hit-rate lift.
A system prompt that includes Current time: ${new Date()}
has a cache hit rate of 0. If the model doesn't actually need the exact time, remove it. If it does, round to the hour or day so cache keys match for a meaningful window.
Put the stable instructions in the system prompt (benefits from provider prefix caching). Put the variable user input in the final user message. This structure is optimal for both types of caching.
If you can't see your hit rate, you can't improve it. Every response in the LLM Gateway dashboard shows cached: true/false
and the hit rate rolls up per model, project, and API key. A hit rate under 10% means caching isn't helping — either the workload is genuinely unique, or your prompts need normalizing.
temperature: 0
on deterministic tasks, and keep timestamps out of prompts.