Eval-Driven Development for a .NET Agent: Replaying Traces Instead of Re-Calling the Model

A .NET developer documented porting Alex Punnen's Python eval-driven development stack, published on Towards AI, to a fully local .NET agent setup using Ollama for both the agent and judge models and a stock OpenTelemetry Collector in Docker writing traces to a plain JSON lines file. The three-layer contract separates live deterministic runs (agent and tool calls repeated N times, minutes per run), zero-model-call trace replay (disk read and parsing, seconds per run), and a sampled LLM judge that makes judge model calls only on a sample. The author notes trace replay cannot show whether a new prompt is better because stored traces came from the old prompt, and reports that the judge confidently passed an answer that would have made an incident worse.

The moment I decided to take evals on my .NET agent seriously was not dramatic. I changed one sentence in a system prompt, ran the agent by hand four times, liked what I saw, and shipped it. Two days later I wanted to know whether that change had made anything worse. The only way I had to find out was to run the agent again, against a model, and squint at the output. That is the expensive way to answer a cheap question. Most of what I wanted to know “did it still call the status tool?”, “did it stop looping?”, “does the answer still mention the service name?” was already sitting in traces I had thrown away. A few weeks ago Alex Punnen published a piece on Towards AI called Eval-Driven Development: A Software Engineering Approach to Production-Grade AI Agents , with a companion repo. His stack is Python: Pydantic AI, OpenTelemetry, Grafana Tempo, SeaweedFS and Pydantic Evals. The part that clicked for me was not any single tool. It was the shape: This article is my port of that shape to .NET, kept fully local: Ollama for both the agent and the judge, and a stock OpenTelemetry Collector in Docker writing traces to a plain JSON lines file. I’ll also show the part I am least proud of, which is the judge confidently passing an answer that would have made an incident worse. One note before starting. Everything I describe about Punnen’s setup comes from his article and his repo README. Where I quote numbers from anyone else’s write-up, I label them as theirs. I have not verified them. Here is the summary I wish I had pinned above my desk before I started. The contract sits on top of all three. +---------------------------+-----------------------+-------------+-------------------------------------------+| Layer | Cost per run | Speed | What it catches |+---------------------------+-----------------------+-------------+-------------------------------------------+| 1. Live deterministic | Agent model calls + | Slow | Flaky tool use, loops, missing facts in || repeat N times | tool calls, x N | minutes | the answer, regressions from a NEW prompt || | | | or model version |+---------------------------+-----------------------+-------------+-------------------------------------------+| 2. Trace replay, | Zero model calls. | Fast | Production traffic violating the current || deterministic | Disk read + parsing | seconds | contract, new checks applied to history, || | | | drift in tool-call counts over time |+---------------------------+-----------------------+-------------+-------------------------------------------+| 3. Trace replay, | Judge model calls | Medium | Wrong-but-plausible answers, ignored tool || sampled LLM judge | only, on a sample | | evidence, tone and safety issues that no || | | | string check can see |+---------------------------+-----------------------+-------------+-------------------------------------------+ The thing that took me too long to understand is what layer 2 does not do. Replaying stored traces cannot tell you whether your new prompt is better. The traces were produced by the old prompt. If you want to test a change, you run layer 1. What replay gives you is the ability to ask new questions of old behavior for free, and to check real traffic against your contract without paying for it twice. I’ll come back to this, because I got it wrong at first. Anthropic’s engineering post Demystifying evals for AI agents gave me words for things I was doing badly. A few of their definitions matter here this is their framework from their own blog, and I find it useful, not something I have validated independently : They also make a point that made me uncomfortable: grading the exact path an agent took tends to produce brittle tests, and it is usually better to grade what the agent produced. Min and max tool-call checks are path checks. I kept them anyway, but I treat them as bounds , never as a required sequence. “Called at least one tool and at most five” survives an agent that finds a smarter route. “Called get service status then search runbook" does not. So in their terms: my contract file defines tasks, each live run is a trial, each stored OpenTelemetry trace is a transcript, and the three layers above are code graders layers 1 and 2 plus a sampled model grader layer 3 . The human grader is me, and that turned out to be a design problem of its own. The agent is deliberately small. It is an on-call helper with two tools: one returns the current status of a service, one searches a tiny runbook. It runs on Microsoft.Extensions.AI MEAI with OllamaSharp, so the same code would run against a hosted model by swapping one line. Layout: eval-dotnet/ docker-compose.yml otel-collector.yaml traces/ <- collector writes traces.jsonl here contracts/ incident-agent.v3.json incident-agent.v4.json Shared/ Contract.cs TraceRecord.cs DeterministicChecks.cs EvalAgent/ <- runs the agent, live checks layer 1 EvalReplay/ <- replays traces, judge sampling layers 2 and 3 I started by copying Punnen’s Tempo plus SeaweedFS setup and then simplified. For a regression corpus I wanted something I could grep, diff and commit. The OpenTelemetry Collector's file exporter writes every batch as one line of OTLP JSON, which is exactly that. docker-compose.yml: services: ollama: image: ollama/ollama ports: - "11434:11434" volumes: - ollama:/root/.ollamaotel-collector: image: otel/opentelemetry-collector-contrib:latest pin a version for real use command: "--config=/etc/otelcol/config.yaml" user: "0:0" so it can write to ./traces volumes: - ./otel-collector.yaml:/etc/otelcol/config.yaml:ro - ./traces:/traces ports: - "4317:4317" OTLP gRPC - "4318:4318" OTLP HTTPvolumes: ollama: {} otel-collector.yaml: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318processors: batch: {}exporters: file: path: /traces/traces.jsonl debug: verbosity: basicservice: pipelines: traces: receivers: otlp processors: batch exporters: file, debug Bring it up and pull two models from different families, one for the agent and one for the judge: mkdir -p traces contractsdocker compose up -ddocker exec -it $ docker compose ps -q ollama ollama pull qwen3:8bdocker exec -it $ docker compose ps -q ollama ollama pull llama3.1:8b I use qwen3:8b for the agent because it handles tool calling reasonably at that size, and llama3.1:8b as the judge. Using a judge from a different model family than the agent is my own habit, not something I have measured the benefit of. One trade-off worth naming: Punnen’s Tempo config uses a 24 hour block retention. That is fine for observability. It is not fine for a regression corpus. A trace that disappears tomorrow is not capital. I let the collector write everything, and when a trace is interesting a failure, a weird edge case, a case I hand-labeled I copy that line into a golden/ file in the repo. Those never expire. The contract is the piece I copied most directly. One file holds the agent identity, the version, the model, the system prompt, the cases and their checks. When I change the prompt, I bump the version and create a new file. Old traces carry the version they were produced under, so replay can look up the matching contract. contracts/incident-agent.v3.json: { "agentId": "incident-agent", "version": 3, "model": "qwen3:8b", "judgeModel": "llama3.1:8b", "systemPrompt": "/no think You are an on-call assistant. Always check the current status of a service with get service status before giving advice, and use search runbook for procedures. Be brief. If you do not know, say so.", "judgeDimensions": { "name": "helpfulness", "question": "On a scale of 1 to 5, how helpful and complete is this answer for an on-call engineer? PASS if 4 or higher." } , "cases": { "id": "payments-latency", "input": "payments-api latency alert just fired. What should I do?", "checks": { "contains": "payments-api" , "maxModelRequests": 4, "minToolCalls": 1, "maxToolCalls": 4 } }, { "id": "search-healthy", "input": "Is search-api OK right now?", "checks": { "contains": "search-api" , "maxModelRequests": 3, "minToolCalls": 1, "maxToolCalls": 2 } }, { "id": "unknown-service", "input": "inventory-api is throwing 500s, help", "checks": { "contains": "inventory-api" , "maxModelRequests": 3, "minToolCalls": 1, "maxToolCalls": 3 } } } That judgeDimensions block is the v3 rubric. Remember it. It is the one that failed me. The /no think at the start of the prompt is Qwen3's soft switch to skip the visible reasoning block. Without it, my contains checks were sometimes passing because the service name appeared in the model's thinking rather than in the answer. The same check code has to run against live activities and against JSON read back from disk, so I normalize both into one record. Put these three files in Shared/. Shared/Contract.cs: using System.Text.Json;namespace EvalShared;public sealed record CaseChecks List<string ? Contains, int MaxModelRequests, int MinToolCalls, int MaxToolCalls ;public sealed record EvalCase string Id, string Input, CaseChecks Checks ;public sealed record JudgeDimension string Name, string Question ;public sealed record AgentContract string AgentId, int Version, string Model, string JudgeModel, string SystemPrompt, List<JudgeDimension JudgeDimensions, List<EvalCase Cases { static readonly JsonSerializerOptions Opts = new { PropertyNameCaseInsensitive = true, ReadCommentHandling = JsonCommentHandling.Skip }; public static AgentContract Load string path = JsonSerializer.Deserialize<AgentContract File.ReadAllText path , Opts ?? throw new InvalidOperationException $"Could not read contract {path}" ; public static Dictionary< string AgentId, int Version , AgentContract LoadAll string dir = Directory.GetFiles dir, " .json" .Select Load .ToDictionary c = c.AgentId, c.Version ;} Shared/TraceRecord.cs: using System.Diagnostics;using System.Globalization;using System.Text.Json;namespace EvalShared;public sealed record SpanView string TraceId, string Name, IReadOnlyDictionary<string, string Attrs ;public sealed record TraceRecord string TraceId, string AgentId, int PromptVersion, string CaseId, string Input, string FinalResult, int ModelRequests, IReadOnlyList<string ToolCalls, IReadOnlyList<string ToolResults { // Build one record from all spans of one trace. Returns null if there is no agent root span. public static TraceRecord? Build IEnumerable<SpanView spans { var list = spans.ToList ; var root = list.FirstOrDefault s = s.Attrs.ContainsKey "agent.id" ; if root is null return null; static string Op SpanView s = s.Attrs.GetValueOrDefault "gen ai.operation.name" ?? ""; var tools = list .Where s = Op s == "execute tool" || s.Name.StartsWith "execute tool", StringComparison.Ordinal .ToList ; return new TraceRecord TraceId: root.TraceId, AgentId: root.Attrs "agent.id" , PromptVersion: int.Parse root.Attrs.GetValueOrDefault "agent.prompt version" ?? "0", CultureInfo.InvariantCulture , CaseId: root.Attrs.GetValueOrDefault "agent.case id" ?? "", Input: root.Attrs.GetValueOrDefault "agent.input" ?? "", FinalResult: root.Attrs.GetValueOrDefault "agent.final result" ?? "", ModelRequests: list.Count s = Op s == "chat" , ToolCalls: tools.Select t = t.Attrs.GetValueOrDefault "gen ai.tool.name" ?? t.Name .ToList , ToolResults: list.Select s = s.Attrs.GetValueOrDefault "agent.tool result" .Where r = r is not null .Cast<string .ToList ; } // Live path: activities captured in memory. public static SpanView FromActivity Activity a = new a.TraceId.ToHexString , a.DisplayName, a.TagObjects.ToDictionary t = t.Key, t = Convert.ToString t.Value, CultureInfo.InvariantCulture ?? "" ; // Replay path: OTLP JSON lines written by the collector's file exporter. public static IEnumerable<SpanView ReadOtlpJsonl string path { foreach var line in File.ReadLines path { if string.IsNullOrWhiteSpace line continue; using var doc = JsonDocument.Parse line ; foreach var rs in Arr doc.RootElement, "resourceSpans" foreach var ss in Arr rs, "scopeSpans" foreach var sp in Arr ss, "spans" { var attrs = new Dictionary<string, string ; foreach var kv in Arr sp, "attributes" attrs kv.GetProperty "key" .GetString = AnyValue kv.GetProperty "value" ; yield return new SpanView sp.GetProperty "traceId" .GetString , sp.GetProperty "name" .GetString , attrs ; } } } static IEnumerable<JsonElement Arr JsonElement e, string name = e.TryGetProperty name, out var a && a.ValueKind == JsonValueKind.Array ? a.EnumerateArray .ToList : Enumerable.Empty<JsonElement ; // OTLP JSON wraps values as {"stringValue": ".."}, {"intValue": "3"}, {"boolValue": true}, ... static string AnyValue JsonElement v { foreach var p in v.EnumerateObject return p.Value.ValueKind == JsonValueKind.String ? p.Value.GetString : p.Value.GetRawText ; return ""; }} Shared/DeterministicChecks.cs: namespace EvalShared;public sealed record CheckResult string Name, bool Passed, string Detail ;public static class DeterministicChecks{ public static List<CheckResult Run TraceRecord t, CaseChecks c { var results = new List<CheckResult ; var text = t.FinalResult; foreach var s in c.Contains ?? results.Add new $"contains '{s}'", text.Contains s, StringComparison.OrdinalIgnoreCase , "" ; if c.MaxModelRequests 0 results.Add new "max model requests", t.ModelRequests <= c.MaxModelRequests, $"{t.ModelRequests} <= {c.MaxModelRequests}" ; results.Add new "min tool calls", t.ToolCalls.Count = c.MinToolCalls, $"{t.ToolCalls.Count} = {c.MinToolCalls}" ; if c.MaxToolCalls 0 results.Add new "max tool calls", t.ToolCalls.Count <= c.MaxToolCalls, $"{t.ToolCalls.Count} <= {c.MaxToolCalls}" ; return results; }} Punnen mentions in his README that Pydantic Evals shipped a MaxToolCalls evaluator but no minimum, so he wrote MinToolCalls himself. I ended up writing both, because in C there was nothing to inherit either way. The minimum is the more useful of the two for me. The most common failure on a small local model is not looping, it is answering from vibes without calling any tool. Create the console app and reference the shared files: dotnet new console -n EvalAgentcd EvalAgentdotnet add package Microsoft.Extensions.AIdotnet add package OllamaSharpdotnet add package OpenTelemetrydotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocoldotnet add package OpenTelemetry.Exporter.InMemory Add this to EvalAgent.csproj inside the <Project element so both apps compile the same check code: <ItemGroup <Compile Include="../Shared/ .cs" / </ItemGroup EvalAgent/IncidentTools.cs: using System.ComponentModel;using System.Diagnostics;public static class IncidentTools{ static readonly Dictionary<string, string Status = new StringComparer.OrdinalIgnoreCase { "payments-api" = "DEGRADED: primary database failover in progress, started 14:02 UTC. Owner: payments-oncall.", "search-api" = "HEALTHY: all checks passing." }; static readonly string Runbook = "payments-api high latency: 1 check get service status. 2 If NO failover is in progress, restart the api pods. 3 NEVER restart during an active database failover, it extends the outage. Wait and post in payments-incident.", "search-api errors: check index freshness, then the query node pool.", "Unknown service: escalate to the platform on-call, do not guess an owner." ; Description "Gets the current health status of a service by name." public static string GetServiceStatus Description "Service name, for example payments-api" string service { var result = Status.TryGetValue service, out var s ? s : $"UNKNOWN: no service named '{service}' is registered."; Activity.Current?.SetTag "agent.tool result", $"get service status {service} = {result}" ; return result; } Description "Searches the on-call runbook and returns matching entries." public static string SearchRunbook Description "Free text query" string query { var hits = Runbook.Where r = query.Split ' ', StringSplitOptions.RemoveEmptyEntries .Any w = r.Contains w, StringComparison.OrdinalIgnoreCase .ToList ; var result = hits.Count == 0 ? "No runbook entries found." : string.Join "\n", hits ; Activity.Current?.SetTag "agent.tool result", $"search runbook {query} = {result}" ; return result; }} The agent.tool result tag matters later. I added it after the judge incident, so the judge could see the evidence the agent saw. Without it, the stored trace only has the final answer, and a judge reading only the final answer is grading prose, not correctness. EvalAgent/Program.cs: using System.Diagnostics;using EvalShared;using Microsoft.Extensions.AI;using OllamaSharp;using OpenTelemetry;using OpenTelemetry.Resources;using OpenTelemetry.Trace;const string SourceName = "EvalAgent";var contractPath = args.ElementAtOrDefault 0 ?? "../contracts/incident-agent.v3.json";var repeat = int.TryParse args.ElementAtOrDefault 1 , out var r ? r : 3;var contract = AgentContract.Load contractPath ;var captured = new List<Activity ;using var tracerProvider = Sdk.CreateTracerProviderBuilder .SetResourceBuilder ResourceBuilder.CreateDefault .AddService contract.AgentId .AddSource SourceName .AddInMemoryExporter captured // layer 1 checks read these .AddOtlpExporter // localhost:4317, ends up in traces/traces.jsonl .Build ;var source = new ActivitySource SourceName ;IChatClient chat = new ChatClientBuilder new OllamaApiClient new Uri "http://localhost:11434" , contract.Model .UseFunctionInvocation configure: f = f.MaximumIterationsPerRequest = 6 .UseOpenTelemetry sourceName: SourceName .Build ;var tools = new List<AITool { AIFunctionFactory.Create IncidentTools.GetServiceStatus, "get service status" , AIFunctionFactory.Create IncidentTools.SearchRunbook, "search runbook" };var failures = 0;foreach var evalCase in contract.Cases for var trial = 1; trial <= repeat; trial++ { captured.Clear ; using var root = source.StartActivity $"invoke agent {contract.AgentId}" { root?.SetTag "gen ai.operation.name", "invoke agent" ; root?.SetTag "agent.id", contract.AgentId ; root?.SetTag "agent.prompt version", contract.Version ; root?.SetTag "agent.case id", evalCase.Id ; root?.SetTag "agent.input", evalCase.Input ; var response = await chat.GetResponseAsync new ChatMessage ChatRole.System, contract.SystemPrompt , new ChatMessage ChatRole.User, evalCase.Input , new ChatOptions { Tools = tools, Temperature = 0.2f } ; root?.SetTag "agent.final result", response.Text ; } var trace = TraceRecord.Build captured.Select TraceRecord.FromActivity ; if trace is null { Console.WriteLine $" {evalCase.Id} {trial} no root span captured" ; failures++; continue; } var results = DeterministicChecks.Run trace, evalCase.Checks ; var ok = results.All x = x.Passed ; if ok failures++; Console.WriteLine $" { ok ? "PASS" : "FAIL" } {evalCase.Id} trial {trial} trace={trace.TraceId} " + $"model requests={trace.ModelRequests} tools= {string.Join ",", trace.ToolCalls } " ; foreach var x in results.Where x = x.Passed Console.WriteLine $" x {x.Name} {x.Detail}" ;}tracerProvider.ForceFlush ;Console.WriteLine failures == 0 ? "All live trials passed." : $"{failures} live trial s failed." ;return failures == 0 ? 0 : 1; Run it: dotnet run -- ../contracts/incident-agent.v3.json 3 Two details that cost me time: The exit code makes this usable as a CI gate, same as in Punnen’s setup. On my machine, three trials of three cases took MY RUN: wall-clock time for 9 live trials with qwen3:8b. That number is the whole argument for layer 2. The replay app reads traces.jsonl or any golden file , groups spans by trace id, rebuilds the same TraceRecord, looks up the contract that matches the trace's agent id and prompt version, and runs the exact same DeterministicChecks. No Ollama call happens unless you ask for judge sampling. dotnet new console -n EvalReplaycd EvalReplaydotnet add package Microsoft.Extensions.AIdotnet add package OllamaSharp Add the same <Compile Include="../Shared/ .cs" / item group to EvalReplay.csproj. EvalReplay/Program.cs: using EvalShared;using Microsoft.Extensions.AI;using OllamaSharp;string Arg string name, string fallback { var i = Array.IndexOf args, name ; return i = 0 && i + 1 < args.Length ? args i + 1 : fallback;}var tracesPath = Arg "--traces", "../traces/traces.jsonl" ;var contractsDir = Arg "--contracts", "../contracts" ;var againstCurrent = args.Contains "--against-current" ;var judgePercent = int.Parse Arg "--judge-sample", "0" ;var seed = int.Parse Arg "--seed", "42" ;var contracts = AgentContract.LoadAll contractsDir ;var latest = contracts.Values.GroupBy c = c.AgentId .ToDictionary g = g.Key, g = g.MaxBy c = c.Version ;var traces = TraceRecord.ReadOtlpJsonl tracesPath .GroupBy s = s.TraceId .Select g = TraceRecord.Build g .Where t = t is not null .Cast<TraceRecord .ToList ;Console.WriteLine $"Loaded {traces.Count} agent traces from {tracesPath}" ;var failures = 0;var checkedTraces = new List< TraceRecord Trace, AgentContract Contract ;foreach var t in traces { AgentContract? contract = againstCurrent ? latest.GetValueOrDefault t.AgentId : contracts.GetValueOrDefault t.AgentId, t.PromptVersion ; var evalCase = contract?.Cases.FirstOrDefault c = c.Id == t.CaseId ; if contract is null || evalCase is null { Console.WriteLine $" SKIP {t.TraceId} no contract for {t.AgentId} v{t.PromptVersion} case '{t.CaseId}'" ; continue; } var results = DeterministicChecks.Run t, evalCase.Checks ; var ok = results.All r = r.Passed ; if ok failures++; checkedTraces.Add t, contract ; Console.WriteLine $" { ok ? "PASS" : "FAIL" } {t.TraceId ..12 } {t.CaseId,-18} v{t.PromptVersion} - contract v{contract.Version}" ; foreach var r in results.Where r = r.Passed Console.WriteLine $" x {r.Name} {r.Detail}" ;}if judgePercent 0 && checkedTraces.Count 0 await Judge.RunSampleAsync checkedTraces, judgePercent, seed ;Console.WriteLine failures == 0 ? "Deterministic replay: all passed." : $"Deterministic replay: {failures} failed." ;return failures == 0 ? 0 : 1; every stored trace against the contract version that produced itdotnet run -- --traces ../traces/traces.jsonl every stored trace against the NEWEST contract: "what would my new checks have caught?"dotnet run -- --traces ../traces/traces.jsonl --against-current a committed golden file, in CIdotnet run -- --traces ../golden/incident-agent.jsonl On my laptop, replaying MY RUN: number of stored traces traces took MY RUN: replay wall-clock time , and none of it touched a model. When I first wired this up, I celebrated a green replay after changing the system prompt. Then I realized every trace in the file had been produced by the previous prompt. Of course it was green. I had re-graded old homework. So here is how I use the two modes now: Testing a new prompt is still layer 1. There is no shortcut. Deterministic checks are blind to meaning. “The answer mentions payments-api and the agent called two tools” says nothing about whether the advice was right. That is what the judge is for, and why I sample it instead of running it on everything: it is the only layer with a variable cost and a variable answer. EvalReplay/Judge.cs: using System.Text.Json;using EvalShared;using Microsoft.Extensions.AI;using OllamaSharp;public sealed record JudgeVerdict string Verdict, string Reason ;public static class Judge{ public static async Task RunSampleAsync List< TraceRecord Trace, AgentContract Contract items, int percent, int seed { var rng = new Random seed ; var take = Math.Max 1, int Math.Ceiling items.Count percent / 100.0 ; var sample = items.OrderBy = rng.Next .Take take .ToList ; Console.WriteLine $"\nJudging {sample.Count} of {items.Count} traces {percent}% sample, seed {seed} " ; foreach var trace, contract in sample { IChatClient judge = new OllamaApiClient new Uri "http://localhost:11434" , contract.JudgeModel ; // One isolated call per dimension instead of one call grading everything. foreach var dim in contract.JudgeDimensions { var v = await AskAsync judge, trace, dim ; Console.WriteLine $" JUDGE {v.Verdict,-7} {trace.TraceId ..12 } {trace.CaseId,-18} {dim.Name}: {v.Reason}" ; } } } static async Task<JudgeVerdict AskAsync IChatClient judge, TraceRecord t, JudgeDimension dim { var evidence = t.ToolResults.Count == 0 ? " no tool results recorded " : string.Join "\n", t.ToolResults ; var prompt = $""" You are grading one answer from an on-call assistant. Answer only the question below. USER QUESTION: {t.Input} TOOL RESULTS THE ASSISTANT RECEIVED treat as ground truth : {evidence} ASSISTANT FINAL ANSWER: {t.FinalResult} QUESTION: {dim.Question} Rules: - Length and formatting are not quality. A short correct answer passes. - If the information above is not enough to decide, answer UNKNOWN. Reply with JSON only: {"verdict": "PASS" | "FAIL" | "UNKNOWN", "reason": "<one sentence "} """; var response = await judge.GetResponseAsync prompt, new ChatOptions { Temperature = 0f, ResponseFormat = ChatResponseFormat.Json } ; try { return JsonSerializer.Deserialize<JudgeVerdict response.Text, new JsonSerializerOptions { PropertyNameCaseInsensitive = true } ?? new JudgeVerdict "UNKNOWN", "empty judge response" ; } catch JsonException { return new JudgeVerdict "UNKNOWN", $"unparseable judge output: {response.Text}" ; } }} Run it on a 20 percent sample: dotnet run -- --traces ../traces/traces.jsonl --judge-sample 20 --seed 7 Judge results print but do not change the exit code. That is a deliberate opinion: until I have calibrated a judge against my own labels, it is advisory. A gate built on an uncalibrated judge is a gate that randomly blocks deploys. Punnen is candid in his article about his judge getting things wrong, which is part of why I trusted the rest of it. His specifics are his to tell the article is member-only on Towards AI . Here is mine. The code above is the version after the fix. Under the v3 rubric, the judge prompt only contained the user question and the final answer, and the single dimension was “how helpful and complete is this answer, 1 to 5, pass at 4 or higher.” On the payments-latency case, one trial produced this trimmed : To resolve the payments-api latency:1. Check the status page and dashboards for payments-api.2. Restart the api pods to clear stuck connections.3. Monitor p99 latency for 10 minutes after the restart.4. If latency persists, escalate to payments-oncall. The deterministic checks passed. It contains “payments-api”. It made two tool calls, inside the bounds. The judge gave it 5 out of 5 : clear, actionable, well structured. It was the worst answer in the whole run. The status tool had returned that a database failover was in progress, and the runbook entry it retrieved says, in capitals, never restart during an active failover. The agent called both tools and then ignored them. Following that answer would have extended the outage. Meanwhile, another trial on the same case answered roughly: “A database failover is in progress since 14:02 UTC. Per the runbook, do not restart the pods. Wait and post in payments-incident.” The judge gave it a 3 and called it “too brief, lacks troubleshooting steps.” Failed. So the judge passed the dangerous answer and failed the correct one. I went back and hand-labeled every payments-latency trace I had, and the v3 judge disagreed with me on MY RUN: x of y hand-labeled traces . The pattern was always the same: it rewarded length and structure. Looking back, the judge was not broken. It was answering the question I asked, which was a vague one, with no access to the evidence. Here is what changed: contracts/incident-agent.v4.json only the parts that changed : { "agentId": "incident-agent", "version": 4, "model": "qwen3:8b", "judgeModel": "llama3.1:8b", "systemPrompt": "/no think You are an on-call assistant. Always check the current status of a service with get service status before giving advice, and use search runbook for procedures. If a tool result contradicts a usual procedure, the tool result wins and you must say so. Be brief. If you do not know, say so.", "judgeDimensions": { "name": "grounded", "question": "Is every factual claim about the service's current state supported by the tool results? PASS or FAIL." }, { "name": "safe", "question": "Does the answer avoid recommending any action that the tool results or runbook say must not be taken right now? PASS or FAIL." }, { "name": "actionable", "question": "Does the answer tell the engineer what to do or not do next, even if the right action is to wait? PASS or FAIL." } , "cases": { "id": "payments-latency", "input": "payments-api latency alert just fired. What should I do?", "checks": { "contains": "payments-api", "failover" , "maxModelRequests": 4, "minToolCalls": 2, "maxToolCalls": 4 } } } The other two cases stay as in v3. The deterministic change is small: on this case, a correct answer must mention the failover and must have called both tools. My first attempt was a “must not contain ‘restart’” check. I replayed history with --against-current before shipping it, and it failed the correct answer, because "do not restart the pods" contains "restart". That is exactly the kind of check you want to discover is wrong against old traces, for free, rather than in CI on a Friday. I dropped it and used contains: failover instead. After the change, the v4 judge on the same hand-labeled traces disagreed with me on MY RUN: x of y . Better, not perfect. I still read every judge FAIL by hand, and I still read a few judge PASSes each week, because a judge that only ever agrees with itself is not being checked by anyone. The judge story has an uncomfortable second half. When I hand-labeled traces, I was acting as the human grader, the gold standard the judge is supposed to be calibrated against. But the “never restart during a failover” rule was not something I knew. I learned it from the runbook I had written for the demo, which I had written after asking someone who has actually run payments on-call. The DoorDash Design Team’s Building AI Evals series puts a name on this. Their second part argues that deciding whose judgment counts is part of eval design, not a step that comes before it. They split evaluation questions into fit ask the individual the result was made for , authority ask people with real standing in the domain and boundaries ask a broad community what the AI should do at all . In their own write-up about “best restaurant” search, they describe finding that citywide questions produced noise while neighborhood-level local experts converged, and that how much the experts agreed with each other told them what kind of product to build. Those are their findings about their system. What I took from it is the framing, not the numbers. Mapped onto my small agent: My v3 rubric quietly collapsed all three into “helpfulness” and handed it to a model that was, at best, imitating me. That is why it failed. The fix was not only a better prompt. It was asking someone with authority to write the safe dimension, and treating their disagreement with my labels as a signal rather than noise. A while ago I wrote that prompts are consumables and evals are capital. Prompts encode how one model version reacts to particular phrasing, so they decay with every upgrade. Evals encode what correct behavior looks like in your domain, so they compound. Stored traces sharpen that idea. A trace is a free, permanent record of a behavior you already paid for. Every time I add a check, I can run it against every trace I have ever kept. Every time I add a judge dimension, I can sample it over months of history. The golden file in my repo grows by a few lines a week, mostly from failures, and it never costs a model call to re-check. There is a related idea in Fareed Khan’s long write-up on loop engineering techniques: his maker-checker pattern, where a second agent verifies work by running its own tests rather than trusting the first agent’s opinion. I only read part of it it is member-only too , and I am not claiming his results. But the instinct is the same one that made me demote my judge to advisory: prefer a check that runs over an opinion that sounds confident. I want to be honest about the parts I have not figured out: If you take one thing from this: write the contract first, record your own root span with everything you’ll want to check later, and keep the traces. Run live checks when you change the prompt or the model. Replay deterministic checks over stored traces for everything else, because it costs nothing. Sample a judge on top, show it the evidence, ask it narrow yes/no questions, and do not let it gate a deploy until you’ve caught it being wrong at least once. You will. Tags: AI Agents, LLM Evaluation, OpenTelemetry, Dotnet, LLM Engineering, Ollama, Software Testing, Observability Eval-Driven Development for a .NET Agent: Replaying Traces Instead of Re-Calling the Model https://pub.towardsai.net/eval-driven-development-for-a-net-agent-replaying-traces-instead-of-re-calling-the-model-36d11183d98e was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.