{"slug": "prompt-caching-explained-how-to-cut-llm-costs-by-30-99", "title": "Prompt Caching Explained: How to Cut LLM Costs by 30–99%", "summary": "LLM Gateway explains how prompt caching can reduce large language model costs by 30–99% and cut latency to sub-millisecond. The post details exact-match and prefix caching strategies, showing how a customer support bot using GPT-4o could save $8,250 per month. LLM Gateway offers automatic caching with no code changes required.", "body_md": "The cheapest LLM request is the one you don't send. If the same question shows up twice, there's no reason to pay twice — the model's answer hasn't changed, and the user doesn't care where it came from.\n\nThat's all prompt caching is. You store the response the first time, and serve it from memory the next time the same request comes in. Done well, it takes 30–99% off the bill and knocks latency down to sub-millisecond. Done badly, it serves stale or wrong answers.\n\nThis post covers both: what caching is, the two kinds you'll run into, where each works, and how to enable it on LLM Gateway without touching your code.\n\nA cache sits between your app and the LLM provider. When a request comes in:\n\nThe second request for the same question is a Redis lookup. It costs nothing. It returns in under a millisecond.\n\n```\nWithout caching:           With caching (second hit):\n─────────────────          ─────────────────────────\nApp  →  Provider           App  →  Gateway cache\n~800ms, $0.01              <1ms, $0.00\n```\n\nThey both save money. They work differently and stack on each other.\n\nHash the full request, store the full response. Only hits when the request is **identical**: same model, same messages, same temperature, same tools.\n\nCache the *prefix* of a prompt — the system message, tool definitions, and any shared context — and reuse it across requests with different user messages at the end.\n\nThey're complementary. LLM Gateway's exact-match caching catches repeat queries at the edge; provider prefix caching reduces cost on everything else. You don't have to choose.\n\nSay you're running a customer support bot. 50,000 requests/day. Average 2,000 input tokens (system prompt + context), 500 output tokens. On GPT-4o that's:\n\n``` php\nPer request: (2000/1M)*$2.50 + (500/1M)*$10 = $0.005 + $0.005 = $0.010\nDaily:       $500\nMonthly:     $15,000\n```\n\nAdd exact-match caching with a modest 40% hit rate (support questions repeat more than you'd think):\n\n```\nCache hits:   20,000 × $0.000 = $0\nCache misses: 30,000 × $0.010 = $300\nDaily:        $300  (40% saved)\nMonthly:      $9,000  (saving $6,000/month)\n```\n\nNow layer provider prefix caching on the remaining misses — the 2,000-token system prompt is identical across all of them, so ~80% of input tokens get the cached rate:\n\n```\nEffective input cost per miss: roughly halves\nCache misses:  30,000 × ~$0.0075 = $225\nDaily:         $225  (55% saved vs. baseline)\nMonthly:       $6,750  (saving $8,250/month)\n```\n\nThe numbers scale with volume. The cache infrastructure costs the same whether you do 100 requests or 100 million.\n\nCaching is not free lunch. Don't enable it blindly:\n\nRule of thumb: if your prompt sets `temperature: 0`\n\nor the task is factual/deterministic, cache it. Otherwise, don't.\n\nNo code changes required. Three steps in the dashboard:\n\nRequests just work. Cached responses show `cost: 0`\n\nin the usage dashboard so you can measure your hit rate directly.\n\n``` python\nimport OpenAI from \"openai\";\n\nconst client = new OpenAI({\n  baseURL: \"https://api.llmgateway.io/v1\",\n  apiKey: process.env.LLM_GATEWAY_API_KEY,\n});\n\n// No special parameters needed — caching is automatic\n// when enabled at the project level.\nconst response = await client.chat.completions.create({\n  model: \"gpt-4o\",\n  messages: [{ role: \"user\", content: \"Summarize: ...\" }],\n  temperature: 0,\n});\n```\n\nFull docs: [docs.llmgateway.io/features/caching](https://docs.llmgateway.io/features/caching).\n\nThe difference between 10% and 70% hit rates is usually prompt hygiene, not the cache itself.\n\n`temperature: 0`\n\non deterministic work\nClassification, extraction, routing, yes/no decisions — none of these benefit from sampling variation. `temperature: 0`\n\nmaximizes cache hits and produces more reliable outputs anyway.\n\n```\n// Bad: each of these is a unique cache key\n\"what are your hours?\";\n\"What are your hours?\";\n\"what are your hours? \";\n\"What are your hours\";\n\n// Good: normalize once, hit the cache every time\nconst normalized = input\n  .trim()\n  .toLowerCase()\n  .replace(/[?.!]+$/, \"\");\n```\n\nLowercase, trim whitespace, collapse punctuation. Small change, big hit-rate lift.\n\nA system prompt that includes `Current time: ${new Date()}`\n\nhas a cache hit rate of 0. If the model doesn't actually need the exact time, remove it. If it does, round to the hour or day so cache keys match for a meaningful window.\n\nPut the stable instructions in the system prompt (benefits from provider prefix caching). Put the variable user input in the final user message. This structure is optimal for both types of caching.\n\nIf you can't see your hit rate, you can't improve it. Every response in the LLM Gateway dashboard shows `cached: true/false`\n\nand the hit rate rolls up per model, project, and API key. A hit rate under 10% means caching isn't helping — either the workload is genuinely unique, or your prompts need normalizing.\n\n`temperature: 0`\n\non deterministic tasks, and keep timestamps out of prompts.", "url": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-llm-costs-by-30-99", "canonical_source": "https://dev.to/smakosh/prompt-caching-explained-how-to-cut-llm-costs-by-30-99-1kad", "published_at": "2026-07-08 17:09:08+00:00", "updated_at": "2026-07-08 17:11:26.560201+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["LLM Gateway", "GPT-4o", "Redis"], "alternates": {"html": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-llm-costs-by-30-99", "markdown": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-llm-costs-by-30-99.md", "text": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-llm-costs-by-30-99.txt", "jsonld": "https://wpnews.pro/news/prompt-caching-explained-how-to-cut-llm-costs-by-30-99.jsonld"}}