Online Evaluation and Production Monitoring A lesson on online evaluation and production monitoring for agentic AI systems argues that offline evaluation only measures past, curated behavior, while online evaluation must judge live production runs that carry no expected outcome. The lesson describes a three-tier architecture: deterministic real-time checks on every run (stop reason, policy gates, tool error rate, step and token budgets, leak patterns), asynchronous LLM-judge evaluators on a sampled share of traffic (groundedness, resolution, tone and clarity, trajectory sanity), and a review queue that routes low-scoring runs to human labels and back into the golden dataset. It advises that policy gate violations in production should page someone immediately and signal a missing pre-execution guardrail, since after-the-fact detection is not a substitute for blocking the action. · Agentic AI · 7 min read 📋 Prerequisites - Reliability, Cost, and Latency previous lesson 🎯 What You'll Learn - Explain what online evaluation catches that offline evaluation cannot - Build reference-free evaluators that run on sampled production traces - Combine explicit and implicit user signals into a quality monitor - Detect input and behavior drift and alert on meaningful changes - Close the loop by turning production failures into offline test cases Offline Evals Measure the Past Your golden dataset is a snapshot of what you thought users would do. Production is what they actually do, and it keeps changing: new product features generate new questions, a pricing change creates a wave of plan-change requests, a marketing campaign brings in customers who phrase things differently, and a provider quietly updates a model. Offline evaluation tells you whether a change is safe to ship. Online evaluation tells you whether what you shipped is still working. You need both, and you need a pipe between them. What’s Different Online | | Offline | Online | |---|---|---| | Inputs | Curated dataset | Live traffic | | Expected outcome | Known per case | Unknown | | Volume | Hundreds of cases | Thousands to millions of runs | | Cost tolerance | Grade everything | Must sample | | Latency | Minutes are fine | Some checks must be real-time | | Purpose | Decide whether to ship | Detect problems and find new cases | The biggest difference is the second row. Production runs have no expected field. Online evaluators must be reference-free : they judge a run from the run itself. The Online Evaluation Architecture production ──▶ traces ──▶ sampler ──┬──▶ real-time checks every run │ └─▶ alerts ├──▶ async evaluators sampled │ └─▶ quality dashboard └──▶ review queue low score / flagged └─▶ human label ──▶ golden dataset Because Lesson 2 made every run produce a structured trace, the same trace format flows into offline and online evaluation. Many graders work in both places. Three Tiers of Online Checks Tier 1: Real-Time, Every Run Cheap deterministic checks that run inline or immediately after each request. These are the graders from Lesson 5 that don’t need an expectation: - Stop reason max steps, error rate - Policy gates refund limit, cross-account access - Tool error rate - Step count and token count budgets - Leak patterns in the response Policy gate violations in production should page someone , not wait for a dashboard. They’re also a sign that a pre-execution guardrail is missing. Online detection after the fact isn’t a substitute for blocking the action. Tier 2: Async, Sampled LLM judges and heavier analysis on a sample of traffic. Reference-free versions of your offline judges: - Groundedness : does the response match what the tools returned? No expectation needed; the evidence is in the trace. - Resolution : did the conversation end with the user’s issue resolved, deferred, or abandoned? - Tone and clarity : same rubric as offline - Trajectory sanity : redundancy, ordering constraints that apply to every run python import random class OnlineEvaluator: def init self, realtime graders, sampled graders, sample rate=0.1, always sample=None : self.rt, self.sampled = realtime graders, sampled graders self.rate = sample rate self.always = always sample or lambda run: False def process self, run - dict: case = Case id=run.trace.trace id, input=run.trace.spans 0 .input, initial state={}, expected={} reference-free scores = g.grade case, run for g in self.rt if self.always run or random.random < self.rate: scores += g.grade case, run for g in self.sampled return {"trace id": run.trace.trace id, "scores": scores, "flag": any not s.passed for s in scores } Always judge runs that are risky or unusual, regardless of the sample rate always = lambda run: any n == "issue refund" for n, in run.trace.tool calls or run.stopped reason = "final answer" or len run.trace.of kind "tool" 8 Stratified sampling beats uniform sampling. A 10% uniform sample of mostly simple invoice questions spends your judge budget on the least interesting traffic. Always sample money-moving runs, errors, long trajectories, and negative-feedback conversations; sample the routine traffic at a lower rate. Tier 3: Human Review A review queue of flagged and low-scoring runs, plus a small random sample to estimate what the flags are missing . Humans label them with the same rubrics as offline. These labels: 1. Become new golden-dataset cases the feedback loop below 2. Continuously recalibrate the online judges Lesson 6 . If judge-human agreement on the review queue drops, the judge needs attention. User Signals Users tell you how the agent is doing, mostly without meaning to. Explicit signals: thumbs up/down, CSAT ratings, written feedback. High quality but sparse often only a few percent of conversations , and biased toward strong reactions. Implicit signals: far more plentiful, and noisier. | Signal | Likely meaning | |---|---| | User rephrases the same request | Agent misunderstood | | User asks for a human | Agent failed or wasn’t trusted | | User abandons mid-conversation | Frustration, or got what they needed elsewhere | | User reopens a ticket within 48h | Resolution didn’t stick | | Very short conversation ending in “thanks” | Probably success | | Customer contacts support through another channel the same day | Strong failure signal | php def implicit signals conversation, later events - dict: user msgs = m "content" for m in conversation if m "role" == "user" return { "rephrase": any similar a, b 0.85 for a, b in zip user msgs, user msgs 1: , "asked for human": any re.search r"\b human|agent|person|representative \b", m, re.I for m in user msgs , "reopened 48h": later events.get "reopened within hours", 999 <= 48, "turns": len user msgs , } No single implicit signal is reliable. Combined, and validated against human labels on a sample, they make a solid quality proxy that covers 100% of traffic for free. Drift Detection Drift is the gradual change that makes an agent that worked last month fail this month. Monitor three kinds. Input drift. The distribution of user requests changes. Track intent distribution from a classifier or embedding clusters week over week. A new cluster appearing usually means a product change generated questions your dataset doesn’t cover. Behavior drift. The agent’s behavior changes with no code change: usually a provider-side model update, a tool API change, or a data change. Track tool-call distributions, mean steps, refusal rate, escalation rate, and response length. Quality drift. Online judge scores and user signals trend down. A simple, robust detector for categorical distributions is the population stability index PSI : python from math import log def psi expected: dict str, float , actual: dict str, float , eps=1e-4 - float: keys = set expected | set actual return sum actual.get k, eps - expected.get k, eps log actual.get k, eps / expected.get k, eps for k in keys baseline = {"duplicate charge": .22, "refund request": .18, "invoice question": .41, "plan change": .12, "cancellation": .07} this week = {"duplicate charge": .19, "refund request": .16, "invoice question": .30, "plan change": .28, "cancellation": .07} print f"PSI = {psi baseline, this week :.3f}" 0.177 → moderate shift A common rule of thumb: PSI below 0.1 is stable, 0.1–0.25 is a moderate shift worth investigating, and above 0.25 is a significant shift. Here, plan-change requests more than doubled. Check whether the golden dataset covers plan changes well enough, and whether the agent’s pass rate on that slice holds up in production. For continuous metrics steps, latency, judge scores , compare rolling windows against a baseline window and alert on sustained shifts, not single spikes. Agents are noisy; alerting on every blip trains people to ignore alerts. Alerting That People Don’t Ignore | Alert | Trigger | Severity | |---|---|---| | Policy gate violation | Any occurrence | Page | | Error / max-steps rate | Above 2x baseline for 30 min | Page | | Escalation rate | Above baseline + 3 SD for 2h | Ticket | | Online groundedness | 7-day mean drops 5+ pts | Ticket | | Intent PSI | Above 0.25 week over week | Ticket | | Cost per task p95 | Above 1.5x baseline for 1 day | Ticket | Every alert should link to example traces. An alert that says “groundedness dropped” without three example runs gets snoozed. Closing the Loop: Production → Golden Dataset This is the most valuable thing online evaluation does. Every production failure is a test case you didn’t know you needed. flagged run ──▶ human review ──▶ confirmed failure? │ yes ▼ case from trace + scrub PII ▼ human writes expected , tags it ▼ PR to evals/datasets/ reviewed like code ▼ fails on current agent? ── no ──▶ keep anyway regression guard │ yes ▼ fix agent; case now passes Lesson 4’s case from trace does the heavy lifting: it reconstructs fixture state from the trace’s tool outputs, so the production failure becomes a reproducible offline case. Combined with replay mode, you can confirm the fix against the exact sequence of tool outputs that caused the original failure. Track the loop’s health with two numbers: - Cases added from production per week : should be steady, not zero - Time from incident to regression case : should be days, not months Privacy and Retention Online evaluation means storing and reading production conversations. Before you build it: - Scrub PII before traces reach the eval pipeline or a third-party judge - Respect retention limits : eval copies of traces are subject to the same data-retention policy as the originals - Check your judge provider’s data terms before sending customer conversations to an external model - Limit reviewer access to what’s needed, and log who viewed what Key Takeaways - Offline evals decide whether to ship; online evals tell you whether what shipped still works - Production runs have no expected outcome, so online graders must be reference-free - Run checks in three tiers: real-time deterministic, async sampled judges, human review - Stratify sampling toward risky and unusual runs - Combine explicit and implicit user signals, validated against human labels - Monitor input , behavior , and quality drift; alert on sustained shifts with example traces - The most important output is the production-to-dataset loop Exercise 1. Pick six graders from earlier lessons that work reference-free and wire them into an OnlineEvaluator 2. Define a stratified sampling rule for your agent 3. Simulate a week of traffic or replay real traces , and compute intent PSI against a baseline week 4. Take three flagged runs, turn them into golden-dataset cases with case from trace , and confirm each reproduces the failure offline