I Thought the Model Drifted. My Cache Key Was Serving Tuesday.

A developer spent two days debugging an LLM endpoint that returned answers for the wrong prompts, initially blaming model drift and sampling noise. The root cause was a cache key that hashed only the user message, ignoring system prompts, temperature, and other request fields, causing collisions. The developer fixed it by hashing a canonical request document and warns that latency is not a reliable indicator of cache identity.

Have you ever watched an LLM endpoint return a clean answer that belonged to a different prompt entirely? I spent forty-eight hours blaming sampling noise, temperature, and a free model that would not sit still. The request logs looked honest enough, and the health check on the box stayed green the whole time. The bug was quieter than that: a cache key that hashed the user message and ignored everything else that actually changes a completion. I was trying to keep a small eval loop cheap, which is a very ordinary instinct. Free-model access is useful when you want overnight volume without treating every call as precious. I parked a thin HTTP wrapper on a free server, hashed each prompt, and stored the JSON body on disk so retries would not hammer the model. Does that sound reasonable? It did, until two different system prompts started colliding on the same key and I spent a day chasing "nondeterminism" that was just a hash. I ran that wrapper against MonkeyCode's free model access on the free server option because I wanted a boring place to reproduce the cache bug, not a production SLA. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing below depends on a named model, a quota, or a hardware claim. The lesson is the key function, and it still applies if you delete the product name from the stack. The wrapper looked like every weekend cache I have written under time pressure. Incoming POST bodies were reduced to user message , run through hashlib.sha256 , and written under ./cache/<hex .json . A hit returned the file. A miss called the model, then wrote the file. I even logged X-Cache: HIT so future-me would feel scientific. That design has one attractive property and one fatal one. The attractive property is that identical user text becomes free after the first call. The fatal property is that user text is not the request. System prompt, temperature, stop sequences, tool schemas, and even a date injected into the preamble all change the meaning of "the same prompt." If those fields are missing from the key, the disk is not a cache. It is a costume party for old answers. I did what I always do when a free model looks flaky: I assumed the model was the flaky part. Here is the ordered list of dead ends, because the order is the lesson. temperature to 0 in the client and celebrated too early. Hits still disagreed with the live miss I forced with curl .A command that wasted a surprising amount of time: example: this proves latency, not correctness curl -s -D - -o /tmp/a.json -X POST http://127.0.0.1:8080/v1/complete \ -H 'Content-Type: application/json' \ -d '{"user":"summarize this log"}' Latency is a terrible oracle for cache identity. Fast and wrong still prints X-Cache: HIT . The break arrived when I stopped reading answers and started reading filenames. Two requests that I could distinguish in the access log were writing the same 64-character hex. Once you see a collision on disk, the model debate is over. The model never got a chance to drift, because the second call never reached it. I dumped the cache directory and grouped by mtime. Several keys had been overwritten hours apart by payloads that only shared the user sentence. The system prompt had changed. The temperature had changed. In one case I had added a stop list and forgotten it, which is the kind of foot-gun you only notice when you diff the request objects instead of the prose. example: find keys that were overwritten rather than created find ./cache -name ' .json' -printf '%T@ %p\n' | sort -n | tail -n 20 sha256sum ./cache/ .json | awk '{print $1}' | sort | uniq -d If uniq -d prints anything, you do not have a model problem. You have an identity problem. The fix is not "stop caching." Caching completions on a free server is still a good idea when the key is honest. The fix is a canonical request document, stable JSON, and a hash over that document. The test below is meant to be copied. It is labeled as an example because I am showing the method, not a published benchmark. python example cache identity helper — stdlib only from future import annotations import hashlib import json from typing import Any KEY FIELDS = "system", "user", "temperature", "stop", "tools", "response format", "seed", def canonical request payload: dict str, Any - dict str, Any : doc = {} for field in KEY FIELDS: if field not in payload: continue doc field = payload field Stable separators and sorted keys so logically equal dicts hash equal. return json.loads json.dumps doc, sort keys=True, separators= ",", ":" def cache key payload: dict str, Any - str: blob = json.dumps canonical request payload , sort keys=True, separators= ",", ":" .encode "utf-8" return hashlib.sha256 blob .hexdigest def naive key payload: dict str, Any - str: the bug I shipped: hash the user sentence only return hashlib.sha256 payload "user" .encode "utf-8" .hexdigest And the collision test that would have saved the first day: example tests — run with: python -m pytest test cache key.py -q from cache key import cache key, naive key BASE = { "system": "Return JSON with keys ok and reason.", "user": "Is this log line an error?", "temperature": 0, "stop": "\n\n" , "tools": , "response format": "json", "seed": 7, } def test naive key collides when system changes : other = dict BASE other "system" = "Return plain text. Never mention JSON." assert naive key BASE == naive key other documents the bug def test canonical key changes when system changes : other = dict BASE other "system" = "Return plain text. Never mention JSON." assert cache key BASE = cache key other def test canonical key changes when temperature changes : other = dict BASE other "temperature" = 0.7 assert cache key BASE = cache key other def test canonical key is stable under key reordering : shuffled = { "user": BASE "user" , "seed": BASE "seed" , "stop": BASE "stop" , "tools": BASE "tools" , "system": BASE "system" , "response format": BASE "response format" , "temperature": BASE "temperature" , } assert cache key BASE == cache key shuffled If you already have a wrapper, drop naive key next to it and assert it fails. That failing test is the artifact. A passing vibe check on two chatty answers is not. I used this table while I was deciding which fields were "part of the request" and which fields were just telemetry. Steal the table. Do not steal my original key function. | Field | Put it in the key? | Why | |---|---|---| user | Yes | This is the obvious part, and still not sufficient. | system | Yes | Instruction changes are the collision I actually hit. | temperature / top p | Yes | Sampling knobs change the distribution you think you cached. | stop | Yes | Silent truncation later looks like a different model. | tools / function schema | Yes | Tool-capable prompts are not the same request as prose-only. | response format | Yes | JSON-mode versus markdown is a semantic fork. | seed | Yes, if the API honors it | If it is ignored, the key should not pretend otherwise. | request id | No | Identity of the call is not identity of the completion. | Wall clock / Date header | Only if the prompt is time-sensitive | Caching "today's on-call" overnight is how Tuesday leaks. | | Auth token | Never the raw token | Hash a stable tenant id if isolation is required. | | Server hostname | No | That belongs in logs, not in the lookup key. | A useful rule showed up while I filled the table. If changing a field can change the bytes you would accept as correct, it belongs in the key. If changing a field only changes how you observe the call, it belongs in the log line. The incomplete key was the headline, but it was not the only break. Disk on a free server is a place, not a promise. I restarted the process once and still had files, then restarted the box and did not. That is not a model failure either. It is an assumption about persistence that nobody wrote down. I also learned that pretty-printed JSON in the cache file is a gift to diffs and a tax on disk. Compact JSON is enough if the key is right. Pretty JSON is for the autopsy directory, which I now keep separate from the hot cache. Mixing them made grep noisy enough that I missed the collision for several hours. One more break that still annoys me: I logged the user message and not the canonical document. When two hits shared a key, the log could not explain why. After the fix, each write stores key , canonical request , created at , and source hit or miss . The completion body is the payload. The metadata is how you argue with yourself tomorrow. example record shape written next to the completion record = { "key": cache key payload , "canonical": canonical request payload , "source": "miss", "created at": "2026-09-03T12:00:00Z", example timestamp only } Notice the timestamp is labeled as an example. I am not claiming a production incident window. I am claiming that a cache without provenance will waste the next forty-eight hours too. I would still cache. I would still use a free model and a free server for an overnight eval loop, because that is a fine place to practice boring infrastructure. I would not start with the model card, the temperature, or a theory of drift. I would start with two requests that a human can tell apart, and I would demand two keys. The repeatable sequence is short enough to keep on a sticky note. naive key and cache key side by side so the bug has a name. X-Cache: HIT as a performance signal, never as a correctness signal.Would I ship this as a platform cache for end-user traffic? No. This is a lab wrapper. Lab wrappers become production incidents when somebody adds tenants and forgets to put the tenant in the key. That is the same bug with a higher blast radius. This approach does not do semantic caching. Two paraphrases of the same question still miss, and that is intentional. If you want similar-question reuse, you are in embedding territory, and you inherit a different class of wrong answers. I am not selling that upgrade here because I did not run it in this loop. Do not use a disk cache of raw prompts if the prompts contain secrets, customer text, or anything you would not write to an unencrypted notebook. A free server is a convenient scratch box. It is not a confidentiality boundary. Hashing the canonical request protects you from collisions. It does not protect the bodies on disk from whoever can list the directory. Skip this whole pattern if you are scoring model quality. Cache hits will make a flaky model look stable, which is how I lost the first day. Eval traffic should bypass the cache or should key on a run id that never repeats. Caching is for cost control after you trust the request identity, not for manufacturing a reassuring pass rate. Also skip it if your API ignores seed or silently drops unknown fields. A key that includes theater knobs will fragment the cache without buying determinism. Read the live request echo, not the docs you remember from another vendor. The model was not drifting in this incident. The cache was confident, fast, and wrong in a way that still looked like language. That is a nastier outage than a 500, because nobody pages on fluency. If you already have a completion cache, run the naive-versus-canonical tests against last week's code before you tune another sampling knob. What field did your key forget? Mine forgot the system prompt, then the stop list, then the fact that the disk could vanish on reboot. I will repeat the tests. I will not repeat the forty-eight hours of arguing with a hash.