LLMOps Is Supply Chain Work: Pinning the Inputs You Don't Control

A developer argues that LLM failures are usually dependency problems in the retrieval corpus, prompt templates, and tool responses rather than the model itself, and proposes treating content like code with an append-only manifest (corpus.lock) that hashes and trust-tiers every document before it enters the index. The approach pairs a regex-based ingest gate that quarantines documents containing model-directed instructions with per-source trust tiers, so first-party docs can be quoted, vendor docs must be attributed, and user uploads never enter the system prompt.

Suggested tags: ai, llm, devops, python Most LLMOps advice is about the model: pick a snapshot, set a temperature, watch the latency. That framing is comfortable because the model is the one thing you can't debug. The failures I've been pulled into were rarely the model. They were the corpus it read, the prompt template someone edited at 4pm on Friday, and the tool response nobody validated. Those are dependencies, and you are not pinning them. Compare a normal service to an LLM feature: torch==2.4.1 goes in a lockfile. Your retrieval corpus is a directory that gets rsynced by a cron job. updated at column and no diff. gpt-4o-latest can change behavior under you with no changelog. None of these are exotic. All of them are ordinary dependency management that we skipped because the artifacts are text and text feels free to change. Two patterns from the last year worth internalizing, because they generalize: Instructions hidden in help pages. Pages that look like support documentation but contain directives aimed at whatever model reads them. If your retrieval layer indexes third-party or user-editable content, a retrieved document is a user message with extra steps. It arrives inside your system prompt's trust boundary and it is written by someone who is not you. Published statistics engineered to be cited. Publish a plausible-looking number with a memorable framing, and eventually it shows up in generated answers and gets laundered into "facts." This is SEO with a longer feedback loop, and it targets your corpus whether you crawl, license, or accept user uploads. Same failure mode both times: the retrieval corpus is writable by parties outside your trust boundary. Prompt injection gets discussed as a prompt problem. It's an ingestion problem. Filtering the output is a losing game; controlling what enters the index is tractable. The cheapest control I know is an append-only manifest — a lockfile for content. python ingest gate.py — run before any document enters the retrieval index. import hashlib import json import pathlib import re Directives that only make sense if a model, not a human, is reading. INSTRUCTION PATTERNS = re.compile r" ignore all ?previous|you are now|do not recommend|system prompt|" r"when asked about .{0,40} say|recommend ", re.IGNORECASE, def sha256 text: str - str: return hashlib.sha256 text.encode "utf-8" .hexdigest def classify text: str - str: Trust tier decides whether the doc is served, quoted, or quarantined. return "quarantine" if INSTRUCTION PATTERNS.search text else "serve" def ingest path: pathlib.Path, manifest: pathlib.Path - dict: text = path.read text encoding="utf-8" record = { "doc id": path.stem, "sha256": sha256 text , "tier": classify text , "bytes": len text.encode "utf-8" , "source": str path.parent , first-party vs. vendor vs. user upload } Append-only: this file is the lockfile for your corpus. with manifest.open "a", encoding="utf-8" as fh: fh.write json.dumps record + "\n" return record if name == " main ": print ingest pathlib.Path "help page.md" , pathlib.Path "corpus.lock" The regex is not the point — it will miss things, and a determined author will paraphrase. The manifest is the point. Once corpus.lock is in git, a pull request that adds forty quarantine-tier documents from a new domain is visible to a reviewer. A corpus that changes silently is not. Pair it with a per-source trust tier that the retrieval layer enforces: first-party docs can be quoted directly, vendor docs must be attributed inline, user uploads never get inserted into the system prompt at all — only into a delimited user turn. A prompt hash, an eval run id, and a deploy id should be joinable three months later. If you can't answer "which prompt and which corpus produced this answer," you have observability for the wrong system. Practical constraints I'd argue for: There's a visible trend toward agents that reason in a continuous latent space and only emit text at action boundaries. That's a real efficiency win and a real observability loss: the intermediate tokens you used to read are gone. The final answer tells you less than it used to. So log the things that survive: the resolved prompt hash, the retrieved document ids and their trust tiers, every tool call with arguments and status, token counts per span, and the model snapshot string the provider actually returned. OpenTelemetry's GenAI semantic conventions give you attribute names that Langfuse, Phoenix, and most vendors will ingest without a custom adapter. Instrument once at the client wrapper, not at every call site. sha256 , source, and trust tier; the manifest is in git.