I Thought the Model Drifted. My Cache Key Was Serving Tuesday.

Have you ever watched an LLM endpoint return a clean answer that belonged to a different prompt entirely? I spent forty-eight hours blaming sampling noise, temperature, and a free model that would not sit still. The request logs looked honest enough, and the health check on the box stayed green the whole time. The bug was quieter than that: a cache key that hashed the user message and ignored everything else that actually changes a completion.

I was trying to keep a small eval loop cheap, which is a very ordinary instinct. Free-model access is useful when you want overnight volume without treating every call as precious. I parked a thin HTTP wrapper on a free server, hashed each prompt, and stored the JSON body on disk so retries would not hammer the model. Does that sound reasonable? It did, until two different system prompts started colliding on the same key and I spent a day chasing "nondeterminism" that was just a hash.

I ran that wrapper against MonkeyCode's free model access on the free server option because I wanted a boring place to reproduce the cache bug, not a production SLA. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing below depends on a named model, a quota, or a hardware claim. The lesson is the key function, and it still applies if you delete the product name from the stack.

The wrapper looked like every weekend cache I have written under time pressure. Incoming POST bodies were reduced to user_message

, run through hashlib.sha256

, and written under ./cache/<hex>.json

. A hit returned the file. A miss called the model, then wrote the file. I even logged X-Cache: HIT

so future-me would feel scientific.

That design has one attractive property and one fatal one. The attractive property is that identical user text becomes free after the first call. The fatal property is that user text is not the request. System prompt, temperature, stop sequences, tool schemas, and even a date injected into the preamble all change the meaning of "the same prompt." If those fields are missing from the key, the disk is not a cache. It is a costume party for old answers.

I did what I always do when a free model looks flaky: I assumed the model was the flaky part. Here is the ordered list of dead ends, because the order is the lesson.

temperature

to 0

in the client and celebrated too early. Hits still disagreed with the live miss I forced with curl

.A command that wasted a surprising amount of time:

curl -s -D - -o /tmp/a.json -X POST http://127.0.0.1:8080/v1/complete \
  -H 'Content-Type: application/json' \
  -d '{"user":"summarize this log"}'

Latency is a terrible oracle for cache identity. Fast and wrong still prints X-Cache: HIT

.

The break arrived when I stopped reading answers and started reading filenames. Two requests that I could distinguish in the access log were writing the same 64-character hex. Once you see a collision on disk, the model debate is over. The model never got a chance to drift, because the second call never reached it.

I dumped the cache directory and grouped by mtime. Several keys had been overwritten hours apart by payloads that only shared the user sentence. The system prompt had changed. The temperature had changed. In one case I had added a stop

list and forgotten it, which is the kind of foot-gun you only notice when you diff the request objects instead of the prose.

find ./cache -name '*.json' -printf '%T@ %p\n' | sort -n | tail -n 20
sha256sum ./cache/*.json | awk '{print $1}' | sort | uniq -d

If uniq -d

prints anything, you do not have a model problem. You have an identity problem.

The fix is not "stop caching." Caching completions on a free server is still a good idea when the key is honest. The fix is a canonical request document, stable JSON, and a hash over that document. The test below is meant to be copied. It is labeled as an example because I am showing the method, not a published benchmark.

from __future__ import annotations

import hashlib
import json
from typing import Any

KEY_FIELDS = (
    "system",
    "user",
    "temperature",
    "stop",
    "tools",
    "response_format",
    "seed",
)

def canonical_request(payload: dict[str, Any]) -> dict[str, Any]:
    doc = {}
    for field in KEY_FIELDS:
        if field not in payload:
            continue
        doc[field] = payload[field]
    return json.loads(json.dumps(doc, sort_keys=True, separators=(",", ":")))

def cache_key(payload: dict[str, Any]) -> str:
    blob = json.dumps(
        canonical_request(payload), sort_keys=True, separators=(",", ":")
    ).encode("utf-8")
    return hashlib.sha256(blob).hexdigest()

def naive_key(payload: dict[str, Any]) -> str:
    return hashlib.sha256(payload["user"].encode("utf-8")).hexdigest()

And the collision test that would have saved the first day:

from cache_key import cache_key, naive_key

BASE = {
    "system": "Return JSON with keys ok and reason.",
    "user": "Is this log line an error?",
    "temperature": 0,
    "stop": ["\n\n"],
    "tools": [],
    "response_format": "json",
    "seed": 7,
}

def test_naive_key_collides_when_system_changes():
    other = dict(BASE)
    other["system"] = "Return plain text. Never mention JSON."
    assert naive_key(BASE) == naive_key(other)  # documents the bug

def test_canonical_key_changes_when_system_changes():
    other = dict(BASE)
    other["system"] = "Return plain text. Never mention JSON."
    assert cache_key(BASE) != cache_key(other)

def test_canonical_key_changes_when_temperature_changes():
    other = dict(BASE)
    other["temperature"] = 0.7
    assert cache_key(BASE) != cache_key(other)

def test_canonical_key_is_stable_under_key_reordering():
    shuffled = {
        "user": BASE["user"],
        "seed": BASE["seed"],
        "stop": BASE["stop"],
        "tools": BASE["tools"],
        "system": BASE["system"],
        "response_format": BASE["response_format"],
        "temperature": BASE["temperature"],
    }
    assert cache_key(BASE) == cache_key(shuffled)

If you already have a wrapper, drop naive_key

next to it and assert it fails. That failing test is the artifact. A passing vibe check on two chatty answers is not.

I used this table while I was deciding which fields were "part of the request" and which fields were just telemetry. Steal the table. Do not steal my original key function.

A useful rule showed up while I filled the table. If changing a field can change the bytes you would accept as correct, it belongs in the key. If changing a field only changes how you observe the call, it belongs in the log line.

The incomplete key was the headline, but it was not the only break. Disk on a free server is a place, not a promise. I restarted the process once and still had files, then restarted the box and did not. That is not a model failure either. It is an assumption about persistence that nobody wrote down.

I also learned that pretty-printed JSON in the cache file is a gift to diffs and a tax on disk. Compact JSON is enough if the key is right. Pretty JSON is for the autopsy directory, which I now keep separate from the hot cache. Mixing them made grep

noisy enough that I missed the collision for several hours.

One more break that still annoys me: I logged the user message and not the canonical document. When two hits shared a key, the log could not explain why. After the fix, each write stores key

, canonical_request

, created_at

, and source

(hit

or miss

). The completion body is the payload. The metadata is how you argue with yourself tomorrow.

record = {
    "key": cache_key(payload),
    "canonical": canonical_request(payload),
    "source": "miss",
    "created_at": "2026-09-03T12:00:00Z",  # example timestamp only
}

Notice the timestamp is labeled as an example. I am not claiming a production incident window. I am claiming that a cache without provenance will waste the next forty-eight hours too.

I would still cache. I would still use a free model and a free server for an overnight eval loop, because that is a fine place to practice boring infrastructure. I would not start with the model card, the temperature, or a theory of drift. I would start with two requests that a human can tell apart, and I would demand two keys.

The repeatable sequence is short enough to keep on a sticky note.

naive_key

and cache_key

side by side so the bug has a name.X-Cache: HIT

as a performance signal, never as a correctness signal.Would I ship this as a platform cache for end-user traffic? No. This is a lab wrapper. Lab wrappers become production incidents when somebody adds tenants and forgets to put the tenant in the key. That is the same bug with a higher blast radius.

This approach does not do semantic caching. Two paraphrases of the same question still miss, and that is intentional. If you want similar-question reuse, you are in embedding territory, and you inherit a different class of wrong answers. I am not selling that upgrade here because I did not run it in this loop.

Do not use a disk cache of raw prompts if the prompts contain secrets, customer text, or anything you would not write to an unencrypted notebook. A free server is a convenient scratch box. It is not a confidentiality boundary. Hashing the canonical request protects you from collisions. It does not protect the bodies on disk from whoever can list the directory.

Skip this whole pattern if you are scoring model quality. Cache hits will make a flaky model look stable, which is how I lost the first day. Eval traffic should bypass the cache or should key on a run id that never repeats. Caching is for cost control after you trust the request identity, not for manufacturing a reassuring pass rate.

Also skip it if your API ignores seed

or silently drops unknown fields. A key that includes theater knobs will fragment the cache without buying determinism. Read the live request echo, not the docs you remember from another vendor.

The model was not drifting in this incident. The cache was confident, fast, and wrong in a way that still looked like language. That is a nastier outage than a 500, because nobody pages on fluency.

If you already have a completion cache, run the naive-versus-canonical tests against last week's code before you tune another sampling knob. What field did your key forget? Mine forgot the system prompt, then the stop list, then the fact that the disk could vanish on reboot. I will repeat the tests. I will not repeat the forty-eight hours of arguing with a hash.