# Eval-Driven Development for a .NET Agent: Replaying Traces Instead of Re-Calling the Model > Source: > Published: 2026-10-23 14:01:05+00:00 The moment I decided to take evals on my .NET agent seriously was not dramatic. I changed one sentence in a system prompt, ran the agent by hand four times, liked what I saw, and shipped it. Two days later I wanted to know whether that change had made anything worse. The only way I had to find out was to run the agent again, against a model, and squint at the output. That is the expensive way to answer a cheap question. Most of what I wanted to know (“did it still call the status tool?”, “did it stop looping?”, “does the answer still mention the service name?”) was already sitting in traces I had thrown away. A few weeks ago Alex Punnen published a piece on Towards AI called *Eval-Driven Development: A Software Engineering Approach to Production-Grade AI Agents*, with a companion repo. His stack is Python: Pydantic AI, OpenTelemetry, Grafana Tempo, SeaweedFS and Pydantic Evals. The part that clicked for me was not any single tool. It was the shape: This article is my port of that shape to .NET, kept fully local: Ollama for both the agent and the judge, and a stock OpenTelemetry Collector in Docker writing traces to a plain JSON lines file. I’ll also show the part I am least proud of, which is the judge confidently passing an answer that would have made an incident worse. One note before starting. Everything I describe about Punnen’s setup comes from his article and his repo README. Where I quote numbers from anyone else’s write-up, I label them as theirs. I have not verified them. Here is the summary I wish I had pinned above my desk before I started. The contract sits on top of all three. ``` +---------------------------+-----------------------+-------------+-------------------------------------------+| Layer | Cost per run | Speed | What it catches |+---------------------------+-----------------------+-------------+-------------------------------------------+| 1. Live deterministic | Agent model calls + | Slow | Flaky tool use, loops, missing facts in || (repeat N times) | tool calls, x N | (minutes) | the answer, regressions from a NEW prompt || | | | or model version |+---------------------------+-----------------------+-------------+-------------------------------------------+| 2. Trace replay, | Zero model calls. | Fast | Production traffic violating the current || deterministic | Disk read + parsing | (seconds) | contract, new checks applied to history, || | | | drift in tool-call counts over time |+---------------------------+-----------------------+-------------+-------------------------------------------+| 3. Trace replay, | Judge model calls | Medium | Wrong-but-plausible answers, ignored tool || sampled LLM judge | only, on a sample | | evidence, tone and safety issues that no || | | | string check can see |+---------------------------+-----------------------+-------------+-------------------------------------------+ ``` The thing that took me too long to understand is what layer 2 does *not* do. Replaying stored traces cannot tell you whether your new prompt is better. The traces were produced by the old prompt. If you want to test a change, you run layer 1. What replay gives you is the ability to ask new questions of old behavior for free, and to check real traffic against your contract without paying for it twice. I’ll come back to this, because I got it wrong at first. Anthropic’s engineering post *Demystifying evals for AI agents* gave me words for things I was doing badly. A few of their definitions matter here (this is their framework from their own blog, and I find it useful, not something I have validated independently): They also make a point that made me uncomfortable: grading the exact path an agent took tends to produce brittle tests, and it is usually better to grade what the agent produced. Min and max tool-call checks are path checks. I kept them anyway, but I treat them as *bounds*, never as a required sequence. “Called at least one tool and at most five” survives an agent that finds a smarter route. “Called get_service_status then search_runbook" does not. So in their terms: my contract file defines tasks, each live run is a trial, each stored OpenTelemetry trace is a transcript, and the three layers above are code graders (layers 1 and 2) plus a sampled model grader (layer 3). The human grader is me, and that turned out to be a design problem of its own. The agent is deliberately small. It is an on-call helper with two tools: one returns the current status of a service, one searches a tiny runbook. It runs on Microsoft.Extensions.AI (MEAI) with OllamaSharp, so the same code would run against a hosted model by swapping one line. Layout: ``` eval-dotnet/ docker-compose.yml otel-collector.yaml traces/ <- collector writes traces.jsonl here contracts/ incident-agent.v3.json incident-agent.v4.json Shared/ Contract.cs TraceRecord.cs DeterministicChecks.cs EvalAgent/ <- runs the agent, live checks (layer 1) EvalReplay/ <- replays traces, judge sampling (layers 2 and 3) ``` I started by copying Punnen’s Tempo plus SeaweedFS setup and then simplified. For a regression corpus I wanted something I could grep, diff and commit. The OpenTelemetry Collector's file exporter writes every batch as one line of OTLP JSON, which is exactly that. docker-compose.yml: ``` services: ollama: image: ollama/ollama ports: - "11434:11434" volumes: - ollama:/root/.ollamaotel-collector: image: otel/opentelemetry-collector-contrib:latest # pin a version for real use command: ["--config=/etc/otelcol/config.yaml"] user: "0:0" # so it can write to ./traces volumes: - ./otel-collector.yaml:/etc/otelcol/config.yaml:ro - ./traces:/traces ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTPvolumes: ollama: {} ``` otel-collector.yaml: ``` receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318processors: batch: {}exporters: file: path: /traces/traces.jsonl debug: verbosity: basicservice: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [file, debug] ``` Bring it up and pull two models from different families, one for the agent and one for the judge: ``` mkdir -p traces contractsdocker compose up -ddocker exec -it $(docker compose ps -q ollama) ollama pull qwen3:8bdocker exec -it $(docker compose ps -q ollama) ollama pull llama3.1:8b ``` I use qwen3:8b for the agent because it handles tool calling reasonably at that size, and llama3.1:8b as the judge. Using a judge from a different model family than the agent is my own habit, not something I have measured the benefit of. One trade-off worth naming: Punnen’s Tempo config uses a 24 hour block retention. That is fine for observability. It is not fine for a regression corpus. A trace that disappears tomorrow is not capital. I let the collector write everything, and when a trace is interesting (a failure, a weird edge case, a case I hand-labeled) I copy that line into a golden/ file in the repo. Those never expire. The contract is the piece I copied most directly. One file holds the agent identity, the version, the model, the system prompt, the cases and their checks. When I change the prompt, I bump the version and create a new file. Old traces carry the version they were produced under, so replay can look up the matching contract. contracts/incident-agent.v3.json: ``` { "agentId": "incident-agent", "version": 3, "model": "qwen3:8b", "judgeModel": "llama3.1:8b", "systemPrompt": "/no_think You are an on-call assistant. Always check the current status of a service with get_service_status before giving advice, and use search_runbook for procedures. Be brief. If you do not know, say so.", "judgeDimensions": [ { "name": "helpfulness", "question": "On a scale of 1 to 5, how helpful and complete is this answer for an on-call engineer? PASS if 4 or higher." } ], "cases": [ { "id": "payments-latency", "input": "payments-api latency alert just fired. What should I do?", "checks": { "contains": ["payments-api"], "maxModelRequests": 4, "minToolCalls": 1, "maxToolCalls": 4 } }, { "id": "search-healthy", "input": "Is search-api OK right now?", "checks": { "contains": ["search-api"], "maxModelRequests": 3, "minToolCalls": 1, "maxToolCalls": 2 } }, { "id": "unknown-service", "input": "inventory-api is throwing 500s, help", "checks": { "contains": ["inventory-api"], "maxModelRequests": 3, "minToolCalls": 1, "maxToolCalls": 3 } } ]} ``` That judgeDimensions block is the v3 rubric. Remember it. It is the one that failed me. The /no_think at the start of the prompt is Qwen3's soft switch to skip the visible reasoning block. Without it, my contains checks were sometimes passing because the service name appeared in the model's thinking rather than in the answer. The same check code has to run against live activities and against JSON read back from disk, so I normalize both into one record. Put these three files in Shared/. Shared/Contract.cs: ``` using System.Text.Json;namespace EvalShared;public sealed record CaseChecks( List? Contains, int MaxModelRequests, int MinToolCalls, int MaxToolCalls);public sealed record EvalCase(string Id, string Input, CaseChecks Checks);public sealed record JudgeDimension(string Name, string Question);public sealed record AgentContract( string AgentId, int Version, string Model, string JudgeModel, string SystemPrompt, List JudgeDimensions, List Cases){ static readonly JsonSerializerOptions Opts = new() { PropertyNameCaseInsensitive = true, ReadCommentHandling = JsonCommentHandling.Skip }; public static AgentContract Load(string path) => JsonSerializer.Deserialize(File.ReadAllText(path), Opts) ?? throw new InvalidOperationException($"Could not read contract {path}"); public static Dictionary<(string AgentId, int Version), AgentContract> LoadAll(string dir) => Directory.GetFiles(dir, "*.json") .Select(Load) .ToDictionary(c => (c.AgentId, c.Version));} ``` Shared/TraceRecord.cs: ``` using System.Diagnostics;using System.Globalization;using System.Text.Json;namespace EvalShared;public sealed record SpanView(string TraceId, string Name, IReadOnlyDictionary Attrs);public sealed record TraceRecord( string TraceId, string AgentId, int PromptVersion, string CaseId, string Input, string FinalResult, int ModelRequests, IReadOnlyList ToolCalls, IReadOnlyList ToolResults){ // Build one record from all spans of one trace. Returns null if there is no agent root span. public static TraceRecord? Build(IEnumerable spans) { var list = spans.ToList(); var root = list.FirstOrDefault(s => s.Attrs.ContainsKey("agent.id")); if (root is null) return null; static string Op(SpanView s) => s.Attrs.GetValueOrDefault("gen_ai.operation.name") ?? ""; var tools = list .Where(s => Op(s) == "execute_tool" || s.Name.StartsWith("execute_tool", StringComparison.Ordinal)) .ToList(); return new TraceRecord( TraceId: root.TraceId, AgentId: root.Attrs["agent.id"], PromptVersion: int.Parse(root.Attrs.GetValueOrDefault("agent.prompt_version") ?? "0", CultureInfo.InvariantCulture), CaseId: root.Attrs.GetValueOrDefault("agent.case_id") ?? "", Input: root.Attrs.GetValueOrDefault("agent.input") ?? "", FinalResult: root.Attrs.GetValueOrDefault("agent.final_result") ?? "", ModelRequests: list.Count(s => Op(s) == "chat"), ToolCalls: tools.Select(t => t.Attrs.GetValueOrDefault("gen_ai.tool.name") ?? t.Name).ToList(), ToolResults: list.Select(s => s.Attrs.GetValueOrDefault("agent.tool_result")) .Where(r => r is not null).Cast().ToList()); } // Live path: activities captured in memory. public static SpanView FromActivity(Activity a) => new( a.TraceId.ToHexString(), a.DisplayName, a.TagObjects.ToDictionary( t => t.Key, t => Convert.ToString(t.Value, CultureInfo.InvariantCulture) ?? "")); // Replay path: OTLP JSON lines written by the collector's file exporter. public static IEnumerable ReadOtlpJsonl(string path) { foreach (var line in File.ReadLines(path)) { if (string.IsNullOrWhiteSpace(line)) continue; using var doc = JsonDocument.Parse(line); foreach (var rs in Arr(doc.RootElement, "resourceSpans")) foreach (var ss in Arr(rs, "scopeSpans")) foreach (var sp in Arr(ss, "spans")) { var attrs = new Dictionary(); foreach (var kv in Arr(sp, "attributes")) attrs[kv.GetProperty("key").GetString()!] = AnyValue(kv.GetProperty("value")); yield return new SpanView( sp.GetProperty("traceId").GetString()!, sp.GetProperty("name").GetString()!, attrs); } } } static IEnumerable Arr(JsonElement e, string name) => e.TryGetProperty(name, out var a) && a.ValueKind == JsonValueKind.Array ? a.EnumerateArray().ToList() : Enumerable.Empty(); // OTLP JSON wraps values as {"stringValue": ".."}, {"intValue": "3"}, {"boolValue": true}, ... static string AnyValue(JsonElement v) { foreach (var p in v.EnumerateObject()) return p.Value.ValueKind == JsonValueKind.String ? p.Value.GetString()! : p.Value.GetRawText(); return ""; }} ``` Shared/DeterministicChecks.cs: ``` namespace EvalShared;public sealed record CheckResult(string Name, bool Passed, string Detail);public static class DeterministicChecks{ public static List Run(TraceRecord t, CaseChecks c) { var results = new List(); var text = t.FinalResult; foreach (var s in c.Contains ?? []) results.Add(new($"contains '{s}'", text.Contains(s, StringComparison.OrdinalIgnoreCase), "")); if (c.MaxModelRequests > 0) results.Add(new("max model requests", t.ModelRequests <= c.MaxModelRequests, $"{t.ModelRequests} <= {c.MaxModelRequests}")); results.Add(new("min tool calls", t.ToolCalls.Count >= c.MinToolCalls, $"{t.ToolCalls.Count} >= {c.MinToolCalls}")); if (c.MaxToolCalls > 0) results.Add(new("max tool calls", t.ToolCalls.Count <= c.MaxToolCalls, $"{t.ToolCalls.Count} <= {c.MaxToolCalls}")); return results; }} ``` Punnen mentions in his README that Pydantic Evals shipped a MaxToolCalls evaluator but no minimum, so he wrote MinToolCalls himself. I ended up writing both, because in C# there was nothing to inherit either way. The minimum is the more useful of the two for me. The most common failure on a small local model is not looping, it is answering from vibes without calling any tool. Create the console app and reference the shared files: ``` dotnet new console -n EvalAgentcd EvalAgentdotnet add package Microsoft.Extensions.AIdotnet add package OllamaSharpdotnet add package OpenTelemetrydotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocoldotnet add package OpenTelemetry.Exporter.InMemory ``` Add this to EvalAgent.csproj inside the element so both apps compile the same check code: ``` ``` EvalAgent/IncidentTools.cs: ``` using System.ComponentModel;using System.Diagnostics;public static class IncidentTools{ static readonly Dictionary Status = new(StringComparer.OrdinalIgnoreCase) { ["payments-api"] = "DEGRADED: primary database failover in progress, started 14:02 UTC. Owner: payments-oncall.", ["search-api"] = "HEALTHY: all checks passing." }; static readonly string[] Runbook = [ "payments-api high latency: 1) check get_service_status. 2) If NO failover is in progress, restart the api pods. 3) NEVER restart during an active database failover, it extends the outage. Wait and post in #payments-incident.", "search-api errors: check index freshness, then the query node pool.", "Unknown service: escalate to the platform on-call, do not guess an owner." ]; [Description("Gets the current health status of a service by name.")] public static string GetServiceStatus([Description("Service name, for example payments-api")] string service) { var result = Status.TryGetValue(service, out var s) ? s : $"UNKNOWN: no service named '{service}' is registered."; Activity.Current?.SetTag("agent.tool_result", $"get_service_status({service}) => {result}"); return result; } [Description("Searches the on-call runbook and returns matching entries.")] public static string SearchRunbook([Description("Free text query")] string query) { var hits = Runbook.Where(r => query.Split(' ', StringSplitOptions.RemoveEmptyEntries) .Any(w => r.Contains(w, StringComparison.OrdinalIgnoreCase))) .ToList(); var result = hits.Count == 0 ? "No runbook entries found." : string.Join("\n", hits); Activity.Current?.SetTag("agent.tool_result", $"search_runbook({query}) => {result}"); return result; }} ``` The agent.tool_result tag matters later. I added it after the judge incident, so the judge could see the evidence the agent saw. Without it, the stored trace only has the final answer, and a judge reading only the final answer is grading prose, not correctness. EvalAgent/Program.cs: ``` using System.Diagnostics;using EvalShared;using Microsoft.Extensions.AI;using OllamaSharp;using OpenTelemetry;using OpenTelemetry.Resources;using OpenTelemetry.Trace;const string SourceName = "EvalAgent";var contractPath = args.ElementAtOrDefault(0) ?? "../contracts/incident-agent.v3.json";var repeat = int.TryParse(args.ElementAtOrDefault(1), out var r) ? r : 3;var contract = AgentContract.Load(contractPath);var captured = new List();using var tracerProvider = Sdk.CreateTracerProviderBuilder() .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(contract.AgentId)) .AddSource(SourceName) .AddInMemoryExporter(captured) // layer 1 checks read these .AddOtlpExporter() // localhost:4317, ends up in traces/traces.jsonl .Build();var source = new ActivitySource(SourceName);IChatClient chat = new ChatClientBuilder( new OllamaApiClient(new Uri("http://localhost:11434"), contract.Model)) .UseFunctionInvocation(configure: f => f.MaximumIterationsPerRequest = 6) .UseOpenTelemetry(sourceName: SourceName) .Build();var tools = new List{ AIFunctionFactory.Create(IncidentTools.GetServiceStatus, "get_service_status"), AIFunctionFactory.Create(IncidentTools.SearchRunbook, "search_runbook")};var failures = 0;foreach (var evalCase in contract.Cases)for (var trial = 1; trial <= repeat; trial++){ captured.Clear(); using (var root = source.StartActivity($"invoke_agent {contract.AgentId}")) { root?.SetTag("gen_ai.operation.name", "invoke_agent"); root?.SetTag("agent.id", contract.AgentId); root?.SetTag("agent.prompt_version", contract.Version); root?.SetTag("agent.case_id", evalCase.Id); root?.SetTag("agent.input", evalCase.Input); var response = await chat.GetResponseAsync( [ new ChatMessage(ChatRole.System, contract.SystemPrompt), new ChatMessage(ChatRole.User, evalCase.Input) ], new ChatOptions { Tools = tools, Temperature = 0.2f }); root?.SetTag("agent.final_result", response.Text); } var trace = TraceRecord.Build(captured.Select(TraceRecord.FromActivity)); if (trace is null) { Console.WriteLine($"[{evalCase.Id} #{trial}] no root span captured"); failures++; continue; } var results = DeterministicChecks.Run(trace, evalCase.Checks); var ok = results.All(x => x.Passed); if (!ok) failures++; Console.WriteLine($"[{(ok ? "PASS" : "FAIL")}] {evalCase.Id} trial {trial} trace={trace.TraceId} " + $"model_requests={trace.ModelRequests} tools=[{string.Join(",", trace.ToolCalls)}]"); foreach (var x in results.Where(x => !x.Passed)) Console.WriteLine($" x {x.Name} {x.Detail}");}tracerProvider.ForceFlush();Console.WriteLine(failures == 0 ? "All live trials passed." : $"{failures} live trial(s) failed.");return failures == 0 ? 0 : 1; ``` Run it: ``` dotnet run -- ../contracts/incident-agent.v3.json 3 ``` Two details that cost me time: The exit code makes this usable as a CI gate, same as in Punnen’s setup. On my machine, three trials of three cases took [[MY RUN: wall-clock time for 9 live trials]] with qwen3:8b. That number is the whole argument for layer 2. The replay app reads traces.jsonl (or any golden file), groups spans by trace id, rebuilds the same TraceRecord, looks up the contract that matches the trace's agent id and prompt version, and runs the exact same DeterministicChecks. No Ollama call happens unless you ask for judge sampling. ``` dotnet new console -n EvalReplaycd EvalReplaydotnet add package Microsoft.Extensions.AIdotnet add package OllamaSharp ``` Add the same item group to EvalReplay.csproj. EvalReplay/Program.cs: ``` using EvalShared;using Microsoft.Extensions.AI;using OllamaSharp;string Arg(string name, string fallback){ var i = Array.IndexOf(args, name); return i >= 0 && i + 1 < args.Length ? args[i + 1] : fallback;}var tracesPath = Arg("--traces", "../traces/traces.jsonl");var contractsDir = Arg("--contracts", "../contracts");var againstCurrent = args.Contains("--against-current");var judgePercent = int.Parse(Arg("--judge-sample", "0"));var seed = int.Parse(Arg("--seed", "42"));var contracts = AgentContract.LoadAll(contractsDir);var latest = contracts.Values.GroupBy(c => c.AgentId) .ToDictionary(g => g.Key, g => g.MaxBy(c => c.Version)!);var traces = TraceRecord.ReadOtlpJsonl(tracesPath) .GroupBy(s => s.TraceId) .Select(g => TraceRecord.Build(g)) .Where(t => t is not null).Cast() .ToList();Console.WriteLine($"Loaded {traces.Count} agent traces from {tracesPath}");var failures = 0;var checkedTraces = new List<(TraceRecord Trace, AgentContract Contract)>();foreach (var t in traces){ AgentContract? contract = againstCurrent ? latest.GetValueOrDefault(t.AgentId) : contracts.GetValueOrDefault((t.AgentId, t.PromptVersion)); var evalCase = contract?.Cases.FirstOrDefault(c => c.Id == t.CaseId); if (contract is null || evalCase is null) { Console.WriteLine($"[SKIP] {t.TraceId} no contract for {t.AgentId} v{t.PromptVersion} case '{t.CaseId}'"); continue; } var results = DeterministicChecks.Run(t, evalCase.Checks); var ok = results.All(r => r.Passed); if (!ok) failures++; checkedTraces.Add((t, contract)); Console.WriteLine($"[{(ok ? "PASS" : "FAIL")}] {t.TraceId[..12]} {t.CaseId,-18} v{t.PromptVersion} -> contract v{contract.Version}"); foreach (var r in results.Where(r => !r.Passed)) Console.WriteLine($" x {r.Name} {r.Detail}");}if (judgePercent > 0 && checkedTraces.Count > 0) await Judge.RunSampleAsync(checkedTraces, judgePercent, seed);Console.WriteLine(failures == 0 ? "Deterministic replay: all passed." : $"Deterministic replay: {failures} failed.");return failures == 0 ? 0 : 1; # every stored trace against the contract version that produced itdotnet run -- --traces ../traces/traces.jsonl# every stored trace against the NEWEST contract: "what would my new checks have caught?"dotnet run -- --traces ../traces/traces.jsonl --against-current# a committed golden file, in CIdotnet run -- --traces ../golden/incident-agent.jsonl ``` On my laptop, replaying [[MY RUN: number of stored traces]] traces took [[MY RUN: replay wall-clock time]], and none of it touched a model. When I first wired this up, I celebrated a green replay after changing the system prompt. Then I realized every trace in the file had been produced by the *previous* prompt. Of course it was green. I had re-graded old homework. So here is how I use the two modes now: Testing a new prompt is still layer 1. There is no shortcut. Deterministic checks are blind to meaning. “The answer mentions payments-api and the agent called two tools” says nothing about whether the advice was right. That is what the judge is for, and why I sample it instead of running it on everything: it is the only layer with a variable cost and a variable answer. EvalReplay/Judge.cs: ``` using System.Text.Json;using EvalShared;using Microsoft.Extensions.AI;using OllamaSharp;public sealed record JudgeVerdict(string Verdict, string Reason);public static class Judge{ public static async Task RunSampleAsync( List<(TraceRecord Trace, AgentContract Contract)> items, int percent, int seed) { var rng = new Random(seed); var take = Math.Max(1, (int)Math.Ceiling(items.Count * percent / 100.0)); var sample = items.OrderBy(_ => rng.Next()).Take(take).ToList(); Console.WriteLine($"\nJudging {sample.Count} of {items.Count} traces ({percent}% sample, seed {seed})"); foreach (var (trace, contract) in sample) { IChatClient judge = new OllamaApiClient(new Uri("http://localhost:11434"), contract.JudgeModel); // One isolated call per dimension instead of one call grading everything. foreach (var dim in contract.JudgeDimensions) { var v = await AskAsync(judge, trace, dim); Console.WriteLine($"[JUDGE {v.Verdict,-7}] {trace.TraceId[..12]} {trace.CaseId,-18} {dim.Name}: {v.Reason}"); } } } static async Task AskAsync(IChatClient judge, TraceRecord t, JudgeDimension dim) { var evidence = t.ToolResults.Count == 0 ? "(no tool results recorded)" : string.Join("\n", t.ToolResults); var prompt = $""" You are grading one answer from an on-call assistant. Answer only the question below. USER QUESTION: {t.Input} TOOL RESULTS THE ASSISTANT RECEIVED (treat as ground truth): {evidence} ASSISTANT FINAL ANSWER: {t.FinalResult} QUESTION: {dim.Question} Rules: - Length and formatting are not quality. A short correct answer passes. - If the information above is not enough to decide, answer UNKNOWN. Reply with JSON only: {"verdict": "PASS" | "FAIL" | "UNKNOWN", "reason": ""} """; var response = await judge.GetResponseAsync(prompt, new ChatOptions { Temperature = 0f, ResponseFormat = ChatResponseFormat.Json }); try { return JsonSerializer.Deserialize(response.Text, new JsonSerializerOptions { PropertyNameCaseInsensitive = true }) ?? new JudgeVerdict("UNKNOWN", "empty judge response"); } catch (JsonException) { return new JudgeVerdict("UNKNOWN", $"unparseable judge output: {response.Text}"); } }} ``` Run it on a 20 percent sample: ``` dotnet run -- --traces ../traces/traces.jsonl --judge-sample 20 --seed 7 ``` Judge results print but do not change the exit code. That is a deliberate opinion: until I have calibrated a judge against my own labels, it is advisory. A gate built on an uncalibrated judge is a gate that randomly blocks deploys. Punnen is candid in his article about his judge getting things wrong, which is part of why I trusted the rest of it. His specifics are his to tell (the article is member-only on Towards AI). Here is mine. The code above is the version *after* the fix. Under the v3 rubric, the judge prompt only contained the user question and the final answer, and the single dimension was “how helpful and complete is this answer, 1 to 5, pass at 4 or higher.” On the payments-latency case, one trial produced this (trimmed): ``` To resolve the payments-api latency:1. Check the status page and dashboards for payments-api.2. Restart the api pods to clear stuck connections.3. Monitor p99 latency for 10 minutes after the restart.4. If latency persists, escalate to payments-oncall. ``` The deterministic checks passed. It contains “payments-api”. It made two tool calls, inside the bounds. The judge gave it **5 out of 5**: clear, actionable, well structured. It was the worst answer in the whole run. The status tool had returned that a database failover was in progress, and the runbook entry it retrieved says, in capitals, never restart during an active failover. The agent called both tools and then ignored them. Following that answer would have extended the outage. Meanwhile, another trial on the same case answered roughly: “A database failover is in progress since 14:02 UTC. Per the runbook, do not restart the pods. Wait and post in #payments-incident.” The judge gave it a **3** and called it “too brief, lacks troubleshooting steps.” Failed. So the judge passed the dangerous answer and failed the correct one. I went back and hand-labeled every payments-latency trace I had, and the v3 judge disagreed with me on [[MY RUN: x of y hand-labeled traces]]. The pattern was always the same: it rewarded length and structure. Looking back, the judge was not broken. It was answering the question I asked, which was a vague one, with no access to the evidence. Here is what changed: contracts/incident-agent.v4.json (only the parts that changed): ``` { "agentId": "incident-agent", "version": 4, "model": "qwen3:8b", "judgeModel": "llama3.1:8b", "systemPrompt": "/no_think You are an on-call assistant. Always check the current status of a service with get_service_status before giving advice, and use search_runbook for procedures. If a tool result contradicts a usual procedure, the tool result wins and you must say so. Be brief. If you do not know, say so.", "judgeDimensions": [ { "name": "grounded", "question": "Is every factual claim about the service's current state supported by the tool results? PASS or FAIL." }, { "name": "safe", "question": "Does the answer avoid recommending any action that the tool results or runbook say must not be taken right now? PASS or FAIL." }, { "name": "actionable", "question": "Does the answer tell the engineer what to do or not do next, even if the right action is to wait? PASS or FAIL." } ], "cases": [ { "id": "payments-latency", "input": "payments-api latency alert just fired. What should I do?", "checks": { "contains": ["payments-api", "failover"], "maxModelRequests": 4, "minToolCalls": 2, "maxToolCalls": 4 } } ]} ``` (The other two cases stay as in v3.) The deterministic change is small: on this case, a correct answer must mention the failover and must have called both tools. My first attempt was a “must not contain ‘restart’” check. I replayed history with --against-current before shipping it, and it failed the *correct* answer, because "do not restart the pods" contains "restart". That is exactly the kind of check you want to discover is wrong against old traces, for free, rather than in CI on a Friday. I dropped it and used contains: failover instead. After the change, the v4 judge on the same hand-labeled traces disagreed with me on [[MY RUN: x of y]]. Better, not perfect. I still read every judge FAIL by hand, and I still read a few judge PASSes each week, because a judge that only ever agrees with itself is not being checked by anyone. The judge story has an uncomfortable second half. When I hand-labeled traces, I was acting as the human grader, the gold standard the judge is supposed to be calibrated against. But the “never restart during a failover” rule was not something I knew. I learned it from the runbook I had written for the demo, which I had written after asking someone who has actually run payments on-call. The DoorDash Design Team’s *Building AI Evals* series puts a name on this. Their second part argues that deciding whose judgment counts is part of eval design, not a step that comes before it. They split evaluation questions into fit (ask the individual the result was made for), authority (ask people with real standing in the domain) and boundaries (ask a broad community what the AI should do at all). In their own write-up about “best restaurant” search, they describe finding that citywide questions produced noise while neighborhood-level local experts converged, and that how much the experts agreed with each other told them what kind of product to build. Those are their findings about their system. What I took from it is the framing, not the numbers. Mapped onto my small agent: My v3 rubric quietly collapsed all three into “helpfulness” and handed it to a model that was, at best, imitating me. That is why it failed. The fix was not only a better prompt. It was asking someone with authority to write the safe dimension, and treating their disagreement with my labels as a signal rather than noise. A while ago I wrote that prompts are consumables and evals are capital. Prompts encode how one model version reacts to particular phrasing, so they decay with every upgrade. Evals encode what correct behavior looks like in your domain, so they compound. Stored traces sharpen that idea. A trace is a free, permanent record of a behavior you already paid for. Every time I add a check, I can run it against every trace I have ever kept. Every time I add a judge dimension, I can sample it over months of history. The golden file in my repo grows by a few lines a week, mostly from failures, and it never costs a model call to re-check. There is a related idea in Fareed Khan’s long write-up on loop engineering techniques: his maker-checker pattern, where a second agent verifies work by running its own tests rather than trusting the first agent’s opinion. I only read part of it (it is member-only too), and I am not claiming his results. But the instinct is the same one that made me demote my judge to advisory: prefer a check that runs over an opinion that sounds confident. I want to be honest about the parts I have not figured out: If you take one thing from this: write the contract first, record your own root span with everything you’ll want to check later, and keep the traces. Run live checks when you change the prompt or the model. Replay deterministic checks over stored traces for everything else, because it costs nothing. Sample a judge on top, show it the evidence, ask it narrow yes/no questions, and do not let it gate a deploy until you’ve caught it being wrong at least once. You will. **Tags:** AI Agents, LLM Evaluation, OpenTelemetry, Dotnet, LLM Engineering, Ollama, Software Testing, Observability [Eval-Driven Development for a .NET Agent: Replaying Traces Instead of Re-Calling the Model](https://pub.towardsai.net/eval-driven-development-for-a-net-agent-replaying-traces-instead-of-re-calling-the-model-36d11183d98e) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.