How Do You Know When AI Is Wrong? The Case for Evals.

A developer argues that evaluation is the oldest discipline in software applied to a component that broke its central assumption, citing OpenAI's April 2025 GPT-4o sycophancy update and its four-day rollback, whose postmortem admitted the company "didn't have specific deployment evaluations tracking sycophancy." Because LLMs don't return the same output for the same input, the developer proposes replacing conventional assertions with per-property thresholds and minimum success rates, showing a support-agent eval run where an aggregate score of 0.919 masked a 0.667 answered rate and a 0.875 intent-accuracy rate below its 0.90 bar.

Evaluation is not a new discipline invented for a new kind of system. It is the oldest discipline in software meeting a component that broke its central move. In April 2025, OpenAI released an update to GPT-4o and rolled it back just four days later. The model had become noticeably sycophantic. It was too eager to agree. It endorsed questionable decisions, reinforced shaky reasoning, and told people what they wanted to hear when what they really needed was pushback. The postmortem included a line that will sound familiar to anyone who's shipped software: "We also didn't have specific deployment evaluations tracking sycophancy." They weren't checking for it, so it got through. That's not some exotic AI-only failure mode. It's one of the oldest failure modes in software. A behavior mattered, nobody had a test for it, and it escaped into production. The remedy is also familiar: decide what good looks like before release, check for it continuously, and make failure visible enough that you can't ignore it. The only catch is that you can't express this particular check as a conventional unit test. Traditional assertions rely on a simple assumption: assertEquals expected, actual Given the same input, you expect the same output. That's the foundation everything else sits on. LLM's don't make that promise. Ask the same question twice and you may get two different answers. One might satisfy the test. The next might not. Nothing changed except the sampling process. That's the part that breaks. What's interesting is how much survives. You still define success before trusting the component. You still run checks on every change. You still treat failure as a reason to stop and investigate rather than an interesting statistic to file away. Which is why the phrase "evals are the new unit tests" never quite lands for me. They're not unit tests, but they're not really new either. Kent Beck's Test-Driven Development: By Example came out in 2002, and even Beck has often described TDD as something he rediscovered rather than invented. The underlying idea is far older than the method's name: decide what correct behavior looks like before you build the thing. What changed is not the discipline. It is that one of the components in your system stopped being able to give the same answer every time. Not another assertion. A threshold. A named property and a minimum success rate. Here's a recent evaluation run from a support agent I've been using as an experiment: Eval run: 24 scenarios, 3 repeats each PROPERTY RATE BAR N RESULT safety 1.000 1.00 72 PASS gate-outcome 1.000 1.00 33 PASS intent-accuracy 0.875 0.90 72 BELOW BAR groundedness 1.000 0.95 18 PASS answered 0.667 0.90 27 BELOW BAR SUITE FAILED Each property has its own threshold because each one represents a different kind of risk. Safety is set to 1.00 because "mostly safe" isn't a meaningful standard. The same goes for approval gates. If the system bypasses one incorrectly, that's a failure. Intent accuracy is different. A classifier that's right 90% of the time can still be extremely useful. The threshold reflects that reality. Now imagine rolling all of those numbers together. The overall score comes out to 0.919,just under 92%. At first glance, that sounds pretty good. But it's also hiding the fact that roughly one-third of answerable questions never received an answer. This kind of aggregate scores are not very much trustable. Once you start averaging properties together, strong areas can mask weak ones. Fluent answers start compensating for unsafe behavior. High retrieval accuracy starts compensating for unanswered requests. The score looks healthy while specific problems disappear into the average. An aggregate score gives you a headline. The individual properties tell you what's actually broken. The N column matters for the same reason. Groundedness was evaluated over 18 runs, not 72, because groundedness only makes sense when the agent retrieved information. Scoring it across all 72 runs would mean manufacturing judgments for cases where the property never applied. A denominator should describe reality, not improve the metric. The genuinely difficult part of evals isn't that models are mysterious. It's choosing the right test. In July 2023, Lingjiao Chen, Matei Zaharia, and James Zou published How is ChatGPT's behavior changing over time? https://arxiv.org/abs/2307.09009 , comparing March and June releases of GPT-3.5 and GPT-4. One result received most of the attention. On a task involving prime-number identification, GPT-4 reportedly dropped from 97.6% accuracy in March to 2.4% in June. A ninety-five point collapse. The story spread quickly because it seemed to confirm a fear many people already had: that the model had somehow become dramatically worse. Then Arvind Narayanan and Sayash Kapoor looked at the test set. Every example in the benchmark was a prime number. Every single one. That changes the interpretation completely. On a dataset like that, a model can perform well simply by developing a bias toward answering "prime." It doesn't necessarily have to demonstrate much understanding of the underlying task. Narayanan and Kapoor showed that once composite numbers were included, the apparent collapse largely disappeared. What had looked like a sharp decline in reasoning ability was, to a large extent, a change in response preference. The benchmark had measured something real. It just wasn't measuring the thing many people thought it was measuring. That's what makes evals tricky. A badly chosen unit test usually fails loudly. You write the wrong expectation, the test turns red, and sooner or later someone notices. A badly chosen eval does something far worse: it returns a number. A confident, precise, quotable number, with a decimal point in it, that may be measuring something other than the thing you actually care about. The eval can be functioning exactly as designed while the dataset quietly points at the wrong target. If I stripped away all the terminology and tooling, there are three testing principles I'd keep almost unchanged. Define the check before you trust the component. The sycophancy incident is really just the absence of that rule. People had noticed the behavior. Internal discussions had happened. Some testers reportedly felt that something was off. But there was no evaluation that could fail automatically, so concern never became a release blocker. Make failure mechanical. The eval run above exits non-zero. The build fails. Nobody has to hold a meeting to decide whether 0.667 is disappointing enough to matter. That decision was already made when somebody set the threshold at 0.90. The discussion happens once, before deployment, instead of every time a result arrives. Refuse to average, keeping properties separate. If two metrics represent different risks, they deserve different thresholds. Safety should not compensate for accuracy. Accuracy should not compensate for completeness. Completeness should not compensate for policy compliance. The moment you allow one property to pay for another, you've started negotiating with the result instead of evaluating it. That last principle might be the only genuinely new habit in the list. Traditional test suites don't need it because they're already unaveraged. One failing test makes the build fail, regardless of how many passing tests surround it. Evals tempt you with percentages. Percentages feel objective. Executive dashboards love them. They compress complexity into a tidy number that fits in a chart. The problem is that reality doesn't always compress cleanly. Sometimes the most important information isn't the score. What is the one property in your system that you'd refuse to average away? Sources: OpenAI, "Sycophancy in GPT-4o: What happened and what we're doing about it", 29 April 2025, and "Expanding on what we missed with sycophancy", 2 May 2025; quoted via the Georgetown Law Tech Institute brief https://www.law.georgetown.edu/tech-institute/research-insights/insights/tech-brief-ai-sycophancy-openai-2/ reproducing them, because openai.com blocks automated retrieval. Lingjiao Chen, Matei Zaharia and James Zou, "How is ChatGPT's behavior changing over time?" https://arxiv.org/abs/2307.09009 , arXiv:2307.09009, July 2023. The prime-number critique is Arvind Narayanan and Sayash Kapoor's, as reported https://bdtechtalks.com/2023/07/24/chatgpt-capabilities-degrading-study/ and covered in Scientific American https://www.scientificamerican.com/article/yes-ai-models-can-get-worse-over-time/ . Kent Beck, Test-Driven Development: By Example, Addison-Wesley, 2002. Eval figures are from a run of my own suite on 3 September 2026, not a benchmark. Accurate as of 3 September 2026. This essay was drafted with a bit of AI assistance. A human holds editorial responsibility for it.