Monitoring AI Quality: Evaluation Pipelines for Non-Deterministic Products A developer outlined an evaluation pipeline for non-deterministic AI features that generates real outputs, uses a second LLM as a judge to grade them, and alerts teams in Slack when quality drops, with cron jobs extending the checks beyond CI to catch silent model updates. The approach pairs a fixed golden dataset of realistic, anonymized fixtures with three tiers of checks—deterministic format validation, code-based metrics, and LLM-as-a-judge scoring—illustrated through a meeting-assistant summarization feature. The core argument is that AI quality is a distribution rather than a single value, so teams should test properties of good answers instead of exact outputs. Shipping an AI feature is the easy part. Knowing that it still works next month is the hard one. In a traditional application, the same input always gives you the same output. That's why tests work so well: you write expect sum 2, 2 .toBe 4 once, and it protects you forever. AI features break that contract. Send the same input to an LLM twice and you get two different answers. Both may be good. One may be a bit worse. And one day, after a prompt tweak or a silent model update, all of them may be wrong, with no error in the logs. In this article we'll walk through how to monitor the quality of an AI feature: a pipeline that generates real outputs, asks a second model to grade them, and tells the team in Slack when quality drops. Then we'll go one step further and run it on a schedule with cron jobs, so you also catch the problems that don't come from your own deploys. How to read this article: if you're on the product or leadership side, the first sections explain the problem and what to measure, with no code required. If you're a developer, the implementation and the cron examples are in "Building the Pipeline" and "Beyond CI". To keep things concrete, we'll use one example throughout the article: a meeting assistant . It reads the transcript of a recorded call and writes a structured summary: key decisions, action items with owners, and highlights tied to moments in the recording. Each team picks a summary template sales call, standup, job interview, customer support and so on , so the same call should produce a different summary depending on that choice. That's a lot of ways to be almost right. With AI, quality is not a value, it's a distribution. Your feature doesn't produce "the right answer"; it produces a range of answers, and most of them are good. The job is to keep that range where you want it. Think of it like a restaurant chain. The same dish is cooked by different cooks, in different kitchens, with ingredients from different suppliers. Nobody tastes every plate, but someone tastes a sample against the recipe on a regular schedule. That's what an evaluation pipeline does for an AI feature. The tricky part is that quality can change even when nobody touches the code. There are three common sources of change: None of these throws an error. The output just gets a little worse, and your users notice before you do. For an AI feature, that's the real cost: not a crash, but trust . A feature that is great 90% of the time and strange the other 10% teaches users to double-check everything, and then it stops saving them time. An evaluation pipeline gives you three things: You stop checking for the answer and start checking for properties of a good answer. Instead of expect output .toBe ... , you ask: is it well-formed? Does it cover the whole call? Is it faithful to the transcript? Each question needs a different kind of check, and they get more expensive as they get smarter. | Check | What it answers | Cost | Example | |---|---|---|---| | Deterministic | Is the output well-formed? | Free, instant | The summary is valid JSON with every required section | | Metric code | Does it have a measurable property? | Cheap | Highlights are spread across the whole recording | | LLM-as-a-judge | Is it accurate, complete and useful? | One extra LLM call per sample | A 1 to 100 score for each metric you define, like accuracy or clarity | All of them run on top of the same foundation: A golden dataset is a fixed set of realistic inputs that represent how your customers use the feature. For our meeting assistant, that could be 20 fixtures: 15 call transcripts and 5 written meeting notes, covering every summary template. A second set of 5 timestamped recordings is used for the coverage check. Use synthetic or anonymized data, never raw customer content. Because the inputs never change, the scores are comparable over time. If last week's average was higher than today's, something on our side or the provider's side changed. Some qualities can be measured with plain code, no AI needed. A good one for our example is coverage . If the summary only cites moments from the first ten minutes of a one-hour call, the reader gets half the picture. So we split the recording into bins of about four minutes and measure how evenly the summary's timestamps spread across them. A score of 0 means everything is bunched in one place; 1 means perfectly even. For the things code can't measure, like "was this decision really made in the call?", you use a second model as a judge . It receives the source, the generated output and a rubric, and grades the output. The most important part is the rubric, and it should be yours . Start by writing down the metrics that define a good output for your product. For our meeting assistant, they could be: A support bot would have different ones, like tone, policy compliance or whether the issue was solved. Then ask the judge to return a score from 1 to 100 for each metric , with a one-sentence reason for each score. Run it with temperature 0 to keep it as stable as possible. Why a separate score per metric instead of one grade? Because it lets you decide what "good enough" means for each metric on its own, as we'll see in the thresholds. The judge is also an LLM, so treat its scores as measurements with some noise, like a thermometer that can be off by a degree. Before trusting the numbers, compare a few judge scores with a human expert's opinion, and adjust the rubric until they mostly agree. A score is only useful when it's compared against something. With a 1 to 100 score per metric, you set a threshold for each metric , based on how much that metric matters to your product: | Metric | Threshold | Why | |---|---|---| | Accuracy | 90 | An invented decision destroys trust | | Owners | 85 | Action items without an owner don't get done | | Completeness | 80 | A missed item is bad, but easy to add by hand | | Clarity | 70 | Style matters, but less than the facts | These numbers are only an example. Choosing them is a product decision, not an engineering one: a healthcare product may want accuracy at 98, while a brainstorming tool may be fine with 70. Together, the thresholds are the quality level you expect from the feature, written down. There are two ways to apply them, and you'll probably want both: Code metrics, like coverage, work the same way: decide how even the spread must be for your product, and fail the run below that. A natural next step for consistency is to run the same input several times and measure the spread of the scores. A feature with an average of 80 and a small spread is a better product than one with the same average that swings between 60 and 95. The pipeline itself doesn't need to be complex. It's a script that runs the golden set against a real environment, calling the same endpoints the product uses, so it tests the whole stack at once: prompts, model, parsing and response format. Run it as a CI job, like a GitHub Action, every time something that can change quality changes: a prompt, a model version or the code around them. Evaluation pipeline: a prompt or model change, or a scheduled run, runs the golden set; a rubric check and a coverage check run in parallel; the results go to the job summary and to a Slack message that is green only when both checks pass The two checks run side by side, and a failure in either one turns the Slack message red. The core of the pipeline is a Python script, run evals.py . For each fixture it makes two calls to OpenAI: one to generate the summary with the same prompt and model the product uses, and one to ask the judge to grade it. Here is a simplified version: python import asyncio import sys from openai import AsyncOpenAI client = AsyncOpenAI reads OPENAI API KEY from the environment MODEL = "gpt-6-luna" the same model the product uses THRESHOLDS = {"accuracy": 90, "owners": 85, "completeness": 80, "clarity": 70} async def generate summary source : Step 1: generate the summary with the product's own prompt response = await client.chat.completions.create model=MODEL, messages= {"role": "system", "content": SUMMARY PROMPT}, {"role": "user", "content": source}, , return response.choices 0 .message.content async def judge summary source, summary : Step 2: ask the judge for a 1-100 score per metric response = await client.chat.completions.create model=MODEL, reasoning effort="none", temperature only works with reasoning off temperature=0, response format={"type": "json object"}, messages= {"role": "system", "content": JUDGE PROMPT}, {"role": "user", "content": f"Source:\n{source}\n\nSummary:\n{summary}"}, , JudgeResult validates the answer, we'll see it in the next section return JudgeResult.model validate json response.choices 0 .message.content async def process fixture fixture : source = fixture "source" a transcript or written notes summary = await generate summary source return await judge summary source, summary async def run all : fixtures = load fixtures results = await asyncio.gather process fixture f for f in fixtures failed = for metric, threshold in THRESHOLDS.items : average = sum result.scores metric for result in results / len results status = "PASS" if average = threshold else "FAIL" print f"{metric}: {average:.1f} threshold {threshold} {status}" if average < threshold: failed.append metric sys.exit 1 if failed else 0 asyncio.run run all If you read our article on async tasks in arrays https://dev.to/quave/improving-javascript-performance-understanding-async-tasks-in-arrays-4el4 , asyncio.gather will look familiar: it's Python's Promise.all . With the SDK's async client, all fixtures run in parallel. LLM calls are slow, so running them one by one would make the pipeline far slower. SUMMARY PROMPT and MODEL should be exactly what the product uses, and JUDGE PROMPT is the rubric from the next section. To keep the example short, step 1 calls OpenAI directly. If your feature runs behind your own API, call that API instead, so each run also tests what's around the model, like how the prompt is built and how the answer is parsed. A complete version should also keep failed requests apart from low scores. A request that errors is reported by name, and any failure fails the run, because "the AI service returned a 500" and "the AI got worse" are different problems. The judge's prompt lists your metrics and asks for a strict JSON answer. A trimmed version for our example: You are evaluating a meeting summary against the call transcript. Score each metric from 1 to 100 and explain each score in one sentence. - accuracy: every decision and action item really happened in the call - completeness: no decision or action item was left out - owners: every action item has a person responsible for it - clarity: someone who missed the meeting understands the summary Answer only with JSON: {"scores": {"accuracy": <1-100 , ...}, "reasons": {"accuracy": "