Every LLM project I have worked on eventually hits the same wall. Your eval suite and your unit tests both want to call a model, and the moment they do, CI slows to a crawl, costs money, needs secret keys, and flakes whenever the vendor changes a checkpoint. The output is nondeterministic by design, so a "did the pipeline work" test starts failing for reasons that have nothing to do with your code.
The fix I keep reaching for is a deterministic mock provider. It implements the exact same interface as your real providers, returns the same shape of response, and is completely offline. Tests that exercise your scoring, aggregation, and reporting logic never touch a network. In this tutorial I will build one from scratch. The code is grounded in my project evalgate, which ships exactly this, but the technique transfers to any stack.
The whole trick depends on one thing: your real provider and your mock must be interchangeable. So define a narrow interface first and make every adapter conform to it.
export interface ProviderRequest {
model: string;
messages?: { role: string; content: string }[];
prompt?: string;
temperature?: number;
maxTokens?: number;
}
export interface ProviderResponse {
output: string;
latencyMs: number;
usage?: { promptTokens: number; completionTokens: number; totalTokens: number };
costUsd?: number;
model: string;
raw?: unknown;
}
export interface Provider {
readonly name: string;
complete(request: ProviderRequest): Promise<ProviderResponse>;
embed?(text: string): Promise<number[]>;
}
Note that the response carries latency, token usage, and cost, not just text. If your mock only returns a string, every code path that reads usage or costUsd stays untested. Model the full shape.
Requests can arrive as a single prompt or as a list of chat messages. The mock only needs one canonical string to reason about, so normalize first.
export function resolvePrompt(request: ProviderRequest): string {
if (request.prompt !== undefined) return request.prompt;
if (request.messages && request.messages.length > 0) {
return request.messages.map((m) => `${m.role}: ${m.content}`).join("\n");
}
return "";
}
The core promise is that the same input always yields the same output. I derive everything from a hash of the prompt and feed that seed into a small pseudo random generator. FNV-1a plus mulberry32 are both a few lines each and have zero dependencies.
const seed = fnv1a(prompt);
const rand = mulberry32(seed);
const latencyMs = Math.round(5 + rand() * 50); // stable "latency" in 5-55ms
Because the seed comes from the prompt, even the fake latency is reproducible. Run the suite a thousand times and you get identical numbers, which is what makes snapshot style assertions possible.
A useful mock is not random. It is steerable. I give it two levers.
Fixtures are canned responses matched as case insensitive substrings against the prompt. The first match wins. This lets a specific test pin an exact output without caring how the mock works internally.
for (const [needle, response] of Object.entries(this.fixtures)) {
if (prompt.toLowerCase().includes(needle.toLowerCase())) return response;
}
Directives are instruction verbs embedded in the prompt itself, so an example suite can exercise real scorer behavior without any per test wiring. I scan for the last verb: body line and act on it.
switch (verb) {
case "exactly":
case "echo":
case "json":
return payload;
case "uppercase":
return payload.toUpperCase();
case "lowercase":
return payload.toLowerCase();
case "reverse":
return [...payload].reverse().join("");
}
Now a case whose prompt ends with exactly: {"ok": true} deterministically produces that JSON, and your JSON scorer gets a real input to grade. Anything with no fixture and no directive falls back to a stable pseudo response derived from the hash, so it is still reproducible:
const base = `mock response for prompt hash ${fnv1a(prompt)}`;
Here is the feature that earns its keep. A green test suite proves your scorers accept good output. It does not prove they reject bad output. If your exact match scorer has a bug that always returns 1.0, every test passes and you never notice.
So I add a degrade flag that deterministically corrupts the output, simulating a dumber model or a regression.
function corrupt(text: string): string {
if (text.length === 0) return "?";
// Break exact match while keeping partial similarity intact.
return text.slice(0, Math.max(1, text.length - 1)) + " [oops]";
}
Dropping the last character and appending a marker is deliberate. It breaks exact match assertions but keeps enough overlap that a similarity scorer still returns a middling score, not zero. That lets you write a test asserting your pipeline scores lower in degrade mode, which is how you prove the scorers actually discriminate.
Finally, make the mock the default so no configuration means no network. A tiny registry maps provider names to lazy factories.
registry.register("mock", () => new MockProvider(mockOptions));
registry.register("openai", (e) => openai(e));
registry.register("anthropic", (e) => anthropic(e));
Your suite references a provider by name. In CI you point it at mock and everything runs offline, for free, in milliseconds. Locally, flip it to a real provider when you actually want to measure model quality.
A mock provider tests your harness, not your model. It will tell you that your scoring, aggregation, cost accounting, and reporting all work. It will never tell you whether GPT or Claude actually answers your prompts well, because it is not a language model. Deterministic mocks and real model evals are complementary. Run the mock on every commit to protect your plumbing, and run real model evals on a schedule or before a release to measure quality. Confusing the two is how teams end up with a perfectly green suite and a product that gives bad answers.
If you want to see all of this wired together with scorers, a runner, and example suites, the full implementation lives in github.com/AgentPostmortem/evalgate. Steal the pattern and give your test suite its keys back.