How to Build an Eval Plan for an LLM Feature Before You Ship It An engineer outlines a versioned eval plan for LLM features that treats the prompt template, model snapshot, sampling settings, context builder and post-processing as a single system to be tested together. The approach pairs a repo-stored golden set of inputs with critical pass/fail checks, organizes cases by slice, over-samples hard and adversarial inputs, and blocks merges when evals fail. Most LLM features get approved the same way. Someone tries a dozen inputs, the outputs look reasonable, and the PR merges. A few weeks later someone tweaks the prompt, or the provider ships a new model snapshot, and the feature gets a little worse. Nothing fails, because nothing was checking. The fix isn't exotic. It's the same discipline you already apply to normal code: write down what correct looks like, turn it into cases, run them on every change, and block the merge when they fail. The difference is that the assertions are fuzzier, so you have to decide a few things up front that a unit test lets you skip. This is the eval plan we use, adapted for engineers. The running example is a feature that summarizes customer support threads for an agent who's picking up the ticket. You're not evaluating "the model". You're evaluating a system: the prompt template, the model ID, sampling settings, whatever assembles the context retrieval, truncation, thread formatting and any post-processing. Change any of those and your old results stop describing what you ship. So version them together and record them with every run: eval/system.yaml feature: ticket-summary model: