LLMOps Is Supply Chain Work: Pinning the Inputs You Don't Control

Suggested tags: ai, llm, devops, python

Most LLMOps advice is about the model: pick a snapshot, set a temperature, watch the latency. That framing is comfortable because the model is the one thing you can't debug. The failures I've been pulled into were rarely the model. They were the corpus it read, the prompt template someone edited at 4pm on Friday, and the tool response nobody validated. Those are dependencies, and you are not pinning them.

Compare a normal service to an LLM feature:

torch==2.4.1 goes in a lockfile. Your retrieval corpus is a directory that gets rsynced by a cron job.updated_at column and no diff.gpt-4o-latest can change behavior under you with no changelog. None of these are exotic. All of them are ordinary dependency management that we skipped because the artifacts are text and text feels free to change.

Two patterns from the last year worth internalizing, because they generalize:

Instructions hidden in help pages. Pages that look like support documentation but contain directives aimed at whatever model reads them. If your retrieval layer indexes third-party or user-editable content, a retrieved document is a user message with extra steps. It arrives inside your system prompt's trust boundary and it is written by someone who is not you.

Published statistics engineered to be cited. Publish a plausible-looking number with a memorable framing, and eventually it shows up in generated answers and gets laundered into "facts." This is SEO with a longer feedback loop, and it targets your corpus whether you crawl, license, or accept user uploads.

Same failure mode both times: the retrieval corpus is writable by parties outside your trust boundary. Prompt injection gets discussed as a prompt problem. It's an ingestion problem. Filtering the output is a losing game; controlling what enters the index is tractable.

The cheapest control I know is an append-only manifest — a lockfile for content.

import hashlib
import json
import pathlib
import re

INSTRUCTION_PATTERNS = re.compile(
    r"(ignore (all )?previous|you are now|do not recommend|system prompt|"
    r"when asked about .{0,40}(say|recommend))",
    re.IGNORECASE,
)

def sha256(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def classify(text: str) -> str:
    return "quarantine" if INSTRUCTION_PATTERNS.search(text) else "serve"

def ingest(path: pathlib.Path, manifest: pathlib.Path) -> dict:
    text = path.read_text(encoding="utf-8")
    record = {
        "doc_id": path.stem,
        "sha256": sha256(text),
        "tier": classify(text),
        "bytes": len(text.encode("utf-8")),
        "source": str(path.parent),  # first-party vs. vendor vs. user upload
    }
    with manifest.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(record) + "\n")
    return record

if __name__ == "__main__":
    print(ingest(pathlib.Path("help_page.md"), pathlib.Path("corpus.lock")))

The regex is not the point — it will miss things, and a determined author will paraphrase. The manifest is the point. Once corpus.lock is in git, a pull request that adds forty quarantine-tier documents from a new domain is visible to a reviewer. A corpus that changes silently is not.

Pair it with a per-source trust tier that the retrieval layer enforces: first-party docs can be quoted directly, vendor docs must be attributed inline, user uploads never get inserted into the system prompt at all — only into a delimited user turn.

A prompt hash, an eval run id, and a deploy id should be joinable three months later. If you can't answer "which prompt and which corpus produced this answer," you have observability for the wrong system.

Practical constraints I'd argue for:

There's a visible trend toward agents that reason in a continuous latent space and only emit text at action boundaries. That's a real efficiency win and a real observability loss: the intermediate tokens you used to read are gone. The final answer tells you less than it used to.

So log the things that survive: the resolved prompt hash, the retrieved document ids and their trust tiers, every tool call with arguments and status, token counts per span, and the model snapshot string the provider actually returned. OpenTelemetry's GenAI semantic conventions give you attribute names that Langfuse, Phoenix, and most vendors will ingest without a custom adapter. Instrument once at the client wrapper, not at every call site.

sha256, source, and trust tier; the manifest is in git.