You review a 30-line template diff. Production sends 40,000 tokens assembled at request time from six sources, four of which changed this week and none of which appear in the pull request. The diff you approved is not the change you shipped.
This is the part of LLMOps that keeps biting teams after they have evals, tracing, and a gateway in place. The template is versioned in git. The context is assembled by code paths nobody treats as release artifacts.
List what actually goes into a request for a typical RAG-plus-tools system:
Each is a dependency with its own version and its own deploy cadence. The retrieved chunks change when someone reindexes. The tool schema changes when a backend engineer renames a parameter and rewrites the description. Neither triggers a review of the prompt.
So when quality drops on Tuesday and no code shipped, there is no diff to look at. That is the failure this whole post is about.
Compute a content hash over the inputs that can change model behaviour, and log it with every trace:
Store fingerprints always. Store full rendered prompts sampled or redacted, depending on your data rules — full-text retention is a compliance decision, not an observability one. The fingerprint is small, cheap, and answers "what changed" during an incident in one query.
The trade-off: a fingerprint that includes a timestamp, a request id, or a per-user greeting has near-zero collision rate and zero diagnostic value. Hash the parts that affect generation, not the metadata around it. I have watched a team spend a day on "why is our prompt hash changing every request" when the answer was a locale string.
You do not need a framework for this. You need three properties: every section carries a trust level, every section carries an eviction priority, and over-budget on a pinned section is a hard error rather than a silent trim.
from dataclasses import dataclass, field
from hashlib import sha256
import re
@dataclass
class Section:
name: str
text: str
trusted: bool # False for retrieval, tool output, user uploads
priority: int # higher survives truncation
pinned: bool = False # never evict; over budget means fail loudly
INSTRUCTION_LIKE = re.compile(
r"(ignore (all )?(previous|prior) instructions|you are now|system prompt)",
re.IGNORECASE,
)
def screen(text: str) -> str:
return INSTRUCTION_LIKE.sub("[filtered]", text)
def count_tokens(s: str) -> int:
return len(s) // 4
@dataclass
class Assembled:
prompt: str
fingerprint: str
dropped: list[str] = field(default_factory=list)
def assemble(sections, budget, model_id, template_version, snapshot_id):
ordered = sorted(sections, key=lambda s: (s.pinned, s.priority), reverse=True)
kept, dropped, used = [], [], 0
for s in ordered:
body = f"<{s.name} trust={'high' if s.trusted else 'low'}>\n"
body += (s.text if s.trusted else screen(s.text)) + f"\n</{s.name}>"
cost = count_tokens(body)
if used + cost > budget:
if s.pinned:
raise ValueError(f"pinned section {s.name} exceeds budget")
dropped.append(s.name)
continue
kept.append((s, body))
used += cost
pos = {id(s): i for i, s in enumerate(sections)}
kept.sort(key=lambda pair: pos[id(pair[0])])
prompt = "\n\n".join(body for _, body in kept)
h = sha256()
for part in (model_id, template_version, snapshot_id, prompt):
h.update(part.encode())
return Assembled(prompt, h.hexdigest()[:16], dropped)
ctx = assemble(
sections=[
Section("question", "why was my claim denied?", trusted=True, priority=10, pinned=True),
Section("policy_chunk", retrieved_text, trusted=False, priority=5),
Section("profile", tenant_blob, trusted=True, priority=1),
],
budget=12_000,
model_id="claude-sonnet-4",
template_version="support-v14",
snapshot_id=index_build_id,
)
Two details in there are load-bearing. Sections render in original insertion order rather than priority order, because reordering the same content changes model behaviour and makes your eval deltas unattributable. And pinned turns budget pressure into an exception instead of a degraded