{"slug": "how-do-you-know-when-ai-is-wrong-the-case-for-evals", "title": "How Do You Know When AI Is Wrong? The Case for Evals.", "summary": "A developer argues that evaluation is the oldest discipline in software applied to a component that broke its central assumption, citing OpenAI's April 2025 GPT-4o sycophancy update and its four-day rollback, whose postmortem admitted the company \"didn't have specific deployment evaluations tracking sycophancy.\" Because LLMs don't return the same output for the same input, the developer proposes replacing conventional assertions with per-property thresholds and minimum success rates, showing a support-agent eval run where an aggregate score of 0.919 masked a 0.667 answered rate and a 0.875 intent-accuracy rate below its 0.90 bar.", "body_md": "*Evaluation is not a new discipline invented for a new kind of system. It is the oldest discipline in software meeting a component that broke its central move.*\n\nIn April 2025, OpenAI released an update to GPT-4o and rolled it back just four days later. The model had become noticeably sycophantic. It was too eager to agree. It endorsed questionable decisions, reinforced shaky reasoning, and told people what they wanted to hear when what they really needed was pushback.\n\nThe postmortem included a line that will sound familiar to anyone who's shipped software:\n\n\"We also didn't have specific deployment evaluations tracking sycophancy.\"\n\nThey weren't checking for it, so it got through.\n\nThat's not some exotic AI-only failure mode. It's one of the oldest failure modes in software. A behavior mattered, nobody had a test for it, and it escaped into production.\n\nThe remedy is also familiar: decide what good looks like before release, check for it continuously, and make failure visible enough that you can't ignore it.\n\nThe only catch is that you can't express this particular check as a conventional unit test.\n\nTraditional assertions rely on a simple assumption: `assertEquals(expected, actual)`\n\nGiven the same input, you expect the same output. That's the foundation everything else sits on.\n\nLLM's don't make that promise. Ask the same question twice and you may get two different answers. One might satisfy the test. The next might not. Nothing changed except the sampling process.\n\nThat's the part that breaks.\n\nWhat's interesting is how much survives.\n\nYou still define success before trusting the component. You still run checks on every change. You still treat failure as a reason to stop and investigate rather than an interesting statistic to file away.\n\nWhich is why the phrase \"evals are the new unit tests\" never quite lands for me. They're not unit tests, but they're not really new either.\n\nKent Beck's *Test-Driven Development: By Example* came out in 2002, and even Beck has often described TDD as something he **rediscovered** rather than invented. The underlying idea is far older than the method's name: decide what correct behavior looks like before you build the thing. ***What changed is not the discipline. It is that one of the components in your system stopped being able to give the same answer every time.**\n\nNot another assertion. A threshold. A named property and a minimum success rate.\n\nHere's a recent evaluation run from a support agent I've been using as an experiment:\n\n```\nEval run: 24 scenarios, 3 repeats each\n\nPROPERTY               RATE      BAR      N   RESULT\nsafety                1.000     1.00     72   PASS\ngate-outcome          1.000     1.00     33   PASS\nintent-accuracy       0.875     0.90     72   BELOW BAR\ngroundedness          1.000     0.95     18   PASS\nanswered              0.667     0.90     27   BELOW BAR\n\nSUITE FAILED\n```\n\nEach property has its own threshold because each one represents a different kind of risk. Safety is set to 1.00 because \"mostly safe\" isn't a meaningful standard. The same goes for approval gates. If the system bypasses one incorrectly, that's a failure. Intent accuracy is different. A classifier that's right 90% of the time can still be extremely useful. The threshold reflects that reality.\n\nNow imagine rolling all of those numbers together. The overall score comes out to 0.919,just under 92%.\n\nAt first glance, that sounds pretty good. But it's also hiding the fact that roughly one-third of answerable questions never received an answer.\n\nThis kind of aggregate scores are not very much trustable. Once you start averaging properties together, strong areas can mask weak ones. Fluent answers start compensating for unsafe behavior. High retrieval accuracy starts compensating for unanswered requests. The score looks healthy while specific problems disappear into the average.\n\nAn aggregate score gives you a headline. The individual properties tell you what's actually broken.\n\nThe N column matters for the same reason.\n\nGroundedness was evaluated over 18 runs, not 72, because groundedness only makes sense when the agent retrieved information. Scoring it across all 72 runs would mean manufacturing judgments for cases where the property never applied.\n\nA denominator should describe reality, not improve the metric.\n\nThe genuinely difficult part of evals isn't that models are mysterious. It's choosing the right test.\n\nIn July 2023, Lingjiao Chen, Matei Zaharia, and James Zou published [*How is ChatGPT's behavior changing over time?*](https://arxiv.org/abs/2307.09009), comparing March and June releases of GPT-3.5 and GPT-4. One result received most of the attention. On a task involving prime-number identification, GPT-4 reportedly dropped from 97.6% accuracy in March to 2.4% in June.\n\nA ninety-five point collapse. The story spread quickly because it seemed to confirm a fear many people already had: that the model had somehow become dramatically worse.\n\nThen Arvind Narayanan and Sayash Kapoor looked at the test set. Every example in the benchmark was a prime number. Every single one. That changes the interpretation completely.\n\nOn a dataset like that, a model can perform well simply by developing a bias toward answering \"prime.\" It doesn't necessarily have to demonstrate much understanding of the underlying task. Narayanan and Kapoor showed that once composite numbers were included, the apparent collapse largely disappeared. What had looked like a sharp decline in reasoning ability was, to a large extent, a change in response preference.\n\nThe benchmark had measured something real. It just wasn't measuring the thing many people thought it was measuring. That's what makes evals tricky.\n\nA badly chosen unit test usually fails loudly. You write the wrong expectation, the test turns red, and sooner or later someone notices. A badly chosen eval does something far worse: it returns a number. A confident, precise, quotable number, with a decimal point in it, that may be measuring something other than the thing you actually care about.\n\nThe eval can be functioning exactly as designed while the dataset quietly points at the wrong target.\n\nIf I stripped away all the terminology and tooling, there are three testing principles I'd keep almost unchanged.\n\n**Define the check before you trust the component.** The sycophancy incident is really just the absence of that rule. People had noticed the behavior. Internal discussions had happened. Some testers reportedly felt that something was off. But there was no evaluation that could fail automatically, so concern never became a release blocker.\n\n**Make failure mechanical.** The eval run above exits non-zero. The build fails. Nobody has to hold a meeting to decide whether 0.667 is disappointing enough to matter. That decision was already made when somebody set the threshold at 0.90. The discussion happens once, before deployment, instead of every time a result arrives.\n\n**Refuse to average, keeping properties separate.** If two metrics represent different risks, they deserve different thresholds. Safety should not compensate for accuracy. Accuracy should not compensate for completeness. Completeness should not compensate for policy compliance. The moment you allow one property to pay for another, you've started negotiating with the result instead of evaluating it.\n\nThat last principle might be the only genuinely new habit in the list. Traditional test suites don't need it because they're already unaveraged. One failing test makes the build fail, regardless of how many passing tests surround it.\n\nEvals tempt you with percentages. Percentages feel objective. Executive dashboards love them. They compress complexity into a tidy number that fits in a chart.\n\nThe problem is that reality doesn't always compress cleanly. Sometimes the most important information isn't the score.\n\n*What is the one property in your system that you'd refuse to average away?*\n\n*Sources: OpenAI, \"Sycophancy in GPT-4o: What happened and what we're doing about it\", 29 April 2025, and \"Expanding on what we missed with sycophancy\", 2 May 2025; quoted via the [Georgetown Law Tech Institute brief](https://www.law.georgetown.edu/tech-institute/research-insights/insights/tech-brief-ai-sycophancy-openai-2/) reproducing them, because openai.com blocks automated retrieval. Lingjiao Chen, Matei Zaharia and James Zou, [\"How is ChatGPT's behavior changing over time?\"](https://arxiv.org/abs/2307.09009), arXiv:2307.09009, July 2023. The prime-number critique is Arvind Narayanan and Sayash Kapoor's, [as reported](https://bdtechtalks.com/2023/07/24/chatgpt-capabilities-degrading-study/) and covered in [Scientific American](https://www.scientificamerican.com/article/yes-ai-models-can-get-worse-over-time/). Kent Beck, Test-Driven Development: By Example, Addison-Wesley, 2002. Eval figures are from a run of my own suite on 3 September 2026, not a benchmark. Accurate as of 3 September 2026.*\n\n*This essay was drafted with a bit of AI assistance. A human holds editorial responsibility for it.*", "url": "https://wpnews.pro/news/how-do-you-know-when-ai-is-wrong-the-case-for-evals", "canonical_source": "https://dev.to/tonal/how-do-you-know-when-ai-is-wrong-the-case-for-evals-45nb", "published_at": "2026-09-26 17:06:07+00:00", "updated_at": "2026-09-26 17:29:00.878874+00:00", "lang": "en", "topics": ["ai-safety", "large-language-models", "ai-agents", "mlops", "ai-research"], "entities": ["OpenAI", "GPT-4o", "Kent Beck"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-do-you-know-when-ai-is-wrong-the-case-for-evals", "markdown": "https://wpnews.pro/news/how-do-you-know-when-ai-is-wrong-the-case-for-evals.md", "text": "https://wpnews.pro/news/how-do-you-know-when-ai-is-wrong-the-case-for-evals.txt", "jsonld": "https://wpnews.pro/news/how-do-you-know-when-ai-is-wrong-the-case-for-evals.jsonld"}}