# Monitoring AI Quality: Evaluation Pipelines for Non-Deterministic Products > Source: > Published: 2026-10-20 19:12:21+00:00 Shipping an AI feature is the easy part. Knowing that it still works next month is the hard one. In a traditional application, the same input always gives you the same output. That's why tests work so well: you write `expect(sum(2, 2)).toBe(4)` once, and it protects you forever. AI features break that contract. Send the same input to an LLM twice and you get two different answers. Both may be good. One may be a bit worse. And one day, after a prompt tweak or a silent model update, all of them may be wrong, with no error in the logs. In this article we'll walk through how to monitor the quality of an AI feature: a pipeline that generates real outputs, asks a second model to grade them, and tells the team in Slack when quality drops. Then we'll go one step further and run it on a schedule with cron jobs, so you also catch the problems that don't come from your own deploys. **How to read this article:** if you're on the product or leadership side, the first sections explain the problem and what to measure, with no code required. If you're a developer, the implementation and the cron examples are in "Building the Pipeline" and "Beyond CI". To keep things concrete, we'll use one example throughout the article: a **meeting assistant**. It reads the transcript of a recorded call and writes a structured summary: key decisions, action items with owners, and highlights tied to moments in the recording. Each team picks a summary template (sales call, standup, job interview, customer support and so on), so the same call should produce a different summary depending on that choice. That's a lot of ways to be almost right. With AI, quality is not a value, it's a distribution. Your feature doesn't produce "the right answer"; it produces a range of answers, and most of them are good. The job is to keep that range where you want it. Think of it like a restaurant chain. The same dish is cooked by different cooks, in different kitchens, with ingredients from different suppliers. Nobody tastes every plate, but someone tastes a sample against the recipe on a regular schedule. That's what an evaluation pipeline does for an AI feature. The tricky part is that quality can change even when nobody touches the code. There are three common sources of change: None of these throws an error. The output just gets a little worse, and your users notice before you do. For an AI feature, that's the real cost: not a crash, but **trust**. A feature that is great 90% of the time and strange the other 10% teaches users to double-check everything, and then it stops saving them time. An evaluation pipeline gives you three things: You stop checking for *the* answer and start checking for **properties** of a good answer. Instead of `expect(output).toBe(...)`, you ask: is it well-formed? Does it cover the whole call? Is it faithful to the transcript? Each question needs a different kind of check, and they get more expensive as they get smarter. | Check | What it answers | Cost | Example | |---|---|---|---| | Deterministic | Is the output well-formed? | Free, instant | The summary is valid JSON with every required section | | Metric (code) | Does it have a measurable property? | Cheap | Highlights are spread across the whole recording | | LLM-as-a-judge | Is it accurate, complete and useful? | One extra LLM call per sample | A 1 to 100 score for each metric you define, like accuracy or clarity | All of them run on top of the same foundation: A **golden dataset** is a fixed set of realistic inputs that represent how your customers use the feature. For our meeting assistant, that could be 20 fixtures: 15 call transcripts and 5 written meeting notes, covering every summary template. A second set of 5 timestamped recordings is used for the coverage check. Use synthetic or anonymized data, never raw customer content. Because the inputs never change, the scores are comparable over time. If last week's average was higher than today's, something on our side or the provider's side changed. Some qualities can be measured with plain code, no AI needed. A good one for our example is **coverage**. If the summary only cites moments from the first ten minutes of a one-hour call, the reader gets half the picture. So we split the recording into bins of about four minutes and measure how evenly the summary's timestamps spread across them. A score of 0 means everything is bunched in one place; 1 means perfectly even. For the things code can't measure, like "was this decision really made in the call?", you use a second model as a **judge**. It receives the source, the generated output and a rubric, and grades the output. The most important part is the rubric, and it should be **yours**. Start by writing down the metrics that define a good output for your product. For our meeting assistant, they could be: A support bot would have different ones, like tone, policy compliance or whether the issue was solved. Then ask the judge to return a **score from 1 to 100 for each metric**, with a one-sentence reason for each score. Run it with temperature 0 to keep it as stable as possible. Why a separate score per metric instead of one grade? Because it lets you decide what "good enough" means for each metric on its own, as we'll see in the thresholds. The judge is also an LLM, so treat its scores as measurements with some noise, like a thermometer that can be off by a degree. Before trusting the numbers, compare a few judge scores with a human expert's opinion, and adjust the rubric until they mostly agree. A score is only useful when it's compared against something. With a 1 to 100 score per metric, you set a **threshold for each metric**, based on how much that metric matters to your product: | Metric | Threshold | Why | |---|---|---| | Accuracy | 90 | An invented decision destroys trust | | Owners | 85 | Action items without an owner don't get done | | Completeness | 80 | A missed item is bad, but easy to add by hand | | Clarity | 70 | Style matters, but less than the facts | These numbers are only an example. Choosing them is a product decision, not an engineering one: a healthcare product may want accuracy at 98, while a brainstorming tool may be fine with 70. Together, the thresholds are the quality level you expect from the feature, written down. There are two ways to apply them, and you'll probably want both: Code metrics, like coverage, work the same way: decide how even the spread must be for your product, and fail the run below that. A natural next step for **consistency** is to run the same input several times and measure the spread of the scores. A feature with an average of 80 and a small spread is a better product than one with the same average that swings between 60 and 95. The pipeline itself doesn't need to be complex. It's a script that runs the golden set against a real environment, calling the same endpoints the product uses, so it tests the whole stack at once: prompts, model, parsing and response format. Run it as a CI job, like a GitHub Action, every time something that can change quality changes: a prompt, a model version or the code around them. ![Evaluation pipeline: a prompt or model change, or a scheduled run, runs the golden set; a rubric check and a coverage check run in parallel; the results go to the job summary and to a Slack message that is green only when both checks pass] The two checks run side by side, and a failure in either one turns the Slack message red. The core of the pipeline is a Python script, `run_evals.py`. For each fixture it makes two calls to OpenAI: one to generate the summary with the same prompt and model the product uses, and one to ask the judge to grade it. Here is a simplified version: ``` python import asyncio import sys from openai import AsyncOpenAI client = AsyncOpenAI() # reads OPENAI_API_KEY from the environment MODEL = "gpt-6-luna" # the same model the product uses THRESHOLDS = {"accuracy": 90, "owners": 85, "completeness": 80, "clarity": 70} async def generate_summary(source): # Step 1: generate the summary with the product's own prompt response = await client.chat.completions.create( model=MODEL, messages=[ {"role": "system", "content": SUMMARY_PROMPT}, {"role": "user", "content": source}, ], ) return response.choices[0].message.content async def judge_summary(source, summary): # Step 2: ask the judge for a 1-100 score per metric response = await client.chat.completions.create( model=MODEL, reasoning_effort="none", # temperature only works with reasoning off temperature=0, response_format={"type": "json_object"}, messages=[ {"role": "system", "content": JUDGE_PROMPT}, {"role": "user", "content": f"Source:\n{source}\n\nSummary:\n{summary}"}, ], ) # JudgeResult validates the answer, we'll see it in the next section return JudgeResult.model_validate_json(response.choices[0].message.content) async def process_fixture(fixture): source = fixture["source"] # a transcript or written notes summary = await generate_summary(source) return await judge_summary(source, summary) async def run_all(): fixtures = load_fixtures() results = await asyncio.gather(*(process_fixture(f) for f in fixtures)) failed = [] for metric, threshold in THRESHOLDS.items(): average = sum(result.scores[metric] for result in results) / len(results) status = "PASS" if average >= threshold else "FAIL" print(f"{metric}: {average:.1f} (threshold {threshold}) {status}") if average < threshold: failed.append(metric) sys.exit(1 if failed else 0) asyncio.run(run_all()) ``` If you read our article on [async tasks in arrays](https://dev.to/quave/improving-javascript-performance-understanding-async-tasks-in-arrays-4el4), `asyncio.gather` will look familiar: it's Python's `Promise.all`. With the SDK's async client, all fixtures run in parallel. LLM calls are slow, so running them one by one would make the pipeline far slower. `SUMMARY_PROMPT` and `MODEL` should be exactly what the product uses, and `JUDGE_PROMPT` is the rubric from the next section. To keep the example short, step 1 calls OpenAI directly. If your feature runs behind your own API, call that API instead, so each run also tests what's around the model, like how the prompt is built and how the answer is parsed. A complete version should also keep failed requests apart from low scores. A request that errors is reported by name, and any failure fails the run, because "the AI service returned a 500" and "the AI got worse" are different problems. The judge's prompt lists your metrics and asks for a strict JSON answer. A trimmed version for our example: ``` You are evaluating a meeting summary against the call transcript. Score each metric from 1 to 100 and explain each score in one sentence. - accuracy: every decision and action item really happened in the call - completeness: no decision or action item was left out - owners: every action item has a person responsible for it - clarity: someone who missed the meeting understands the summary Answer only with JSON: {"scores": {"accuracy": <1-100>, ...}, "reasons": {"accuracy": "", ...}} ``` Validate the answer before using it. The judge is an LLM too, so sometimes it returns a score out of range, or something that isn't JSON at all. With Pydantic, that becomes an error instead of a made-up number: ``` python from typing import Annotated from pydantic import BaseModel, Field Score = Annotated[int, Field(ge=1, le=100)] class JudgeResult(BaseModel): scores: dict[str, Score] # one 1-100 score per metric reasons: dict[str, str] # one sentence per metric ``` A score per metric tells you **what kind** of problem you have. Low accuracy usually means the AI is inventing things that didn't happen in the call. Low completeness means items were skipped. Low clarity means the summary is vague. Each one points to a different fix, and the reasons tell you where to look. Version the judge's prompt too, just like the prompts it evaluates; a prompt management tool like Langfuse makes that easy. And if you want one headline number for a dashboard, the average of your metrics works well. A check that nobody sees is a check that doesn't exist. Every run publishes its results in three places: Evaluation pipelines are not immune to the problem they solve. They call slow, non-deterministic services, so they time out, the judge occasionally returns something unparseable, and a deploy may not be live yet when the run starts. When that happens often enough, teams tend to switch the automatic run to manual, and a manual check is one that people forget to run. The lesson: **separate "the pipeline broke" from "quality dropped"**. They need different alerts and often different owners. If every red message might be noise, people stop reading them, and then the pipeline protects nobody. A pipeline that runs after a deploy answers one question: *did we break something?* But remember the three sources of change. Two of them, the model provider and your customers' inputs, never come with a deploy. A model can change on a Tuesday afternoon while your team is in a meeting. To catch that, the evaluation has to run on a clock, not on a commit. There are two kinds of scheduled runs, and they answer different questions: | Scheduled run | Inputs | Question it answers | |---|---|---| | Golden set on a cron | The same fixtures every time | Did something change under us? | | Production sampling | A random sample of real outputs | How good is what customers actually get? | The golden set is your **control group**: the inputs never change, so any change in the score comes from the system. Production sampling is your **reality check**: the inputs change every day, and that's the point. The easiest step is to reuse what you already have. Take the same validation script and add a `schedule` trigger: ``` on: schedule: - cron: "0 11 * * 1-5" # weekdays at 11:00 UTC workflow_dispatch: ``` A few things to keep in mind with scheduled GitHub workflows: The golden set tells you if the system changed. It doesn't tell you how the AI is doing with the inputs customers send today. For that, you sample real outputs and grade them with the same judge. Here's what that job can look like in Node, using `node-cron` for the schedule and `p-map` to limit how many judge calls run at once: ``` python import cron from 'node-cron'; import pMap from 'p-map'; const SAMPLE_SIZE = 20; const MAX_DROP = 5; // points below the 7-day average const SIX_HOURS = 6 * 60 * 60 * 1000; const average = (values) => values.reduce((sum, value) => sum + value, 0) / values.length; cron.schedule('0 */6 * * *', async () => { // 1. Sample real outputs generated since the last run const samples = await AiOutputs.aggregate([ { $match: { createdAt: { $gte: new Date(Date.now() - SIX_HOURS) } } }, { $sample: { size: SAMPLE_SIZE } }, ]).toArray(); if (!samples.length) return; // 2. Grade them with the same judge and metrics used in CI, 4 at a time const results = await pMap(samples, (sample) => judge(sample), { concurrency: 4, }); // 3. Store the run, so we can see trends and not only pass/fail const metrics = Object.keys(results[0].scores); const averages = Object.fromEntries( metrics.map((metric) => [ metric, average(results.map((result) => result.scores[metric])), ]) ); await EvalRuns.insertOne({ createdAt: new Date(), averages, promptVersion: samples[0].promptVersion, model: samples[0].model, scores: results.map(({ outputId, scores }) => ({ outputId, scores })), }); // 4. Alert on any metric that dropped below its recent baseline const baseline = await getRollingAverages({ days: 7 }); const dropped = metrics.filter( (metric) => averages[metric] < baseline[metric] - MAX_DROP ); if (dropped.length) { await notifySlack( dropped .map( (metric) => `${metric}: ${averages[metric].toFixed(1)} vs 7-day average ${baseline[metric].toFixed(1)}` ) .join('\n') ); } }); ``` `AiOutputs`, `EvalRuns`, `judge`, `getRollingAverages` and `notifySlack` stand for your own collections and helpers. The shape is what matters: **sample, grade, store, compare**. Some points that are easy to miss: `node-cron` fires on every one of them, and you pay for the same evaluation several times. Use a scheduler that runs once, like a Kubernetes CronJob, or a package with a shared lock, like SyncedCron in Meteor. A fixed threshold catches big drops. But if accuracy goes from 97 to 91, it still passes a threshold of 90, and something clearly got worse. Comparing each run against a rolling baseline, like the 7-day average in the example above, catches the slow slides that a fixed number misses. And remember the lesson from the previous section: give "the job failed to run" its own alert. A monitor that fails silently looks exactly like a monitor that says everything is fine. One number is enough to start, but a few more turn a red light into a diagnosis. These are the metrics we find useful, and who usually reads each one: | Metric | What it tells you | Who reads it | |---|---|---| | Average judge score | Overall quality, the headline number | Product, leadership | | Score vs. 7-day baseline | Whether quality is sliding, even above the threshold | Product, leadership | | Score per scenario (e.g. per summary template) | Where the AI is weak, and which customers feel it | Product | | Score per metric (e.g. accuracy, clarity) | What kind of problem to fix | Engineering | | Spread across repeated runs | Consistency: how much the answer varies for the same input | Product, engineering | | Coverage and other code metrics | Measurable properties the judge can miss | Engineering | | Latency per call | How long users wait for the AI | Product, engineering | | Failed runs and request errors | Whether the monitor itself is healthy | Engineering | | Tokens per run | What the feature costs, and what monitoring it costs | Leadership | Put the numbers where people already look. For leadership, that can be a single daily line in Slack: the average, the trend against the baseline, passed or failed. For engineers, it's the job summary with each metric's score and a link to the generated outputs. The goal is that nobody has to remember to go check a dashboard. AI features don't fail like regular code. They don't crash; they drift. A prompt change, a model update or a new kind of input can make the output a little worse, and no log will tell you. The only way to know is to keep measuring. The good news is that the tools for this are the ones you already use: a test script, a CI workflow, a cron job and a Slack channel. What changes is what you assert. Not the exact answer, but the properties of a good one: well-formed, complete, faithful to the source, and good enough by a rubric that your team agreed on. If you want to start this week, here's the path we'd follow again: Your AI feature will never give the same answer twice. With an evaluation pipeline watching it, it can still give the same quality.