Diff Review Can't See Your Agent Regress. A Standing Suite Can. Per-diff code review cannot detect regressions in AI coding agents, because the changes that break them most — vendor model-alias updates and server-side changes — produce no diff to review, according to an evals-and-verification analysis. The analysis cites research showing that under the same model name, the share of AI-generated code that ran without edits fell from about half to about one in ten over a few months, and that a 2025 server-side problem at an AI provider affected users with no code change on their side. It recommends a standing suite triggered on every agent-system change (prompts, harness, model alias) plus a nightly cron, with capability and regression run as separate suites and every missed failure routed through an intake process to a named owner. Evals & Verification Diff Review Can’t See Your Agent Regress. A Standing Suite Can. 1. Per-diff review only proves one PR looked right; it can’t answer whether your coding agent still does everything it did last quarter, and the vendor and model-alias changes that break it hardest often produce no diff to review at all. 2. The fix is a standing suite triggered on every agent-system change prompts, harness, model alias plus a nightly cron for the vendor-side changes you’ll never see, with capability and regression run as separate suites so a new capability gain can’t hide a lost regression behind one blended number. 3. No eval suite predicts failures in advance, so every miss has to run through an intake process that turns it into a new regression case, and that case needs a named owner or it gets quietly loosened away the next time someone “fixes” a flake. The plain-language edition for readers outside AI engineering no background needed · 10 min This is about why approving every individual update to an AI coding assistant can still leave you blind to the assistant quietly getting worse overall, and what a team should run instead. The big idea Imagine a restaurant where a manager tastes each dish the moment a cook plates it, to check that specific plate looks right. That tells you today’s dish was made correctly. It tells you nothing about whether the kitchen could still make every dish on the menu as well as it did last month, especially if the oven’s temperature has quietly drifted or a supplier swapped an ingredient without telling anyone. To know that, someone has to periodically cook through the whole menu and taste everything, on a schedule, whether or not anyone changed a recipe that week. Reviewing each update to a coding AI is the first kind of check. It tells a team that one change looked fine to the person who read it. It does not tell them whether the AI still does everything it used to do. That second question needs a different kind of check: a standing test suite that runs on its own schedule and checks the whole system, not just the latest change. One change at a time isn’t the same as the whole system A real example from the research: under the exact same model name, the share of code one AI produced that could run without any edits dropped from about half the time to about one in ten, over just a few months. Nobody edited any code to cause that; the likely cause was that a newer version of the AI started tacking extra text onto its answers. Nobody could have caught that by reviewing individual updates, because there was no update to review. The problem was easy to miss precisely because the generated code sits inside a larger pipeline, and an AI coding assistant is exactly that kind of pipeline. Ordinary software ran into the same wall before AI entered the picture. At one point, more than 80% of the pushes to one of Google’s production systems caused bugs that affected real users and had to be undone. The fix wasn’t reading each change more carefully. It was requiring tests that run constantly, on every change. That cut emergency fixes in half within a year. AI systems make the gap worse, because they can change behavior with no code change at all for a person to even look at. Trigger the check on every kind of change, not just swapping the AI model Most teams only think to rerun their checks when they switch to a different AI model. But there are at least four kinds of change that can shift behavior: editing the instructions file the AI reads, changing its toolset, the AI company quietly pointing a “latest version” label at something new, and the AI company changing something on their own servers that a customer never sees at all. The last two never produce anything for a reviewer to look at, the same way a drifting oven temperature never shows up as a memo on anyone’s desk. A real incident backs this up: during a stretch in 2025, a server-side problem at an AI provider affected a meaningful share of requests, and the company’s own checks at the time didn’t catch the degradation users were already reporting. The fix is running two kinds of checks side by side: ones that fire automatically the moment specific files like the instructions file or the toolset change, and a separate check that runs every night against the live system regardless of whether anything visibly changed, because that nightly run is the only thing that will ever catch the vendor-side kind of problem. Keep two separate scorecards, not one blended number Suppose the AI gains a new ability, say, writing database updates, and the team adds tests for that new ability. The overall pass rate can rise at the very moment something else quietly breaks, because a single blended number lets a new win hide a lost ability. The source gives a concrete bad-versus-good comparison: saying “the score went from 81% to 84%, ship it” hides the fact that one older, already-working test started failing the same week. The better version reports two separate numbers: how the new, still-developing ability is progressing expected to start weak and climb, that’s normal , and whether everything the AI already could do still works expected to stay at or near 100%, because any drop there means something broke . A test for a new ability should only graduate into the “must still work” pile once it passes reliably across repeated tries, not the first week it happens to pass. And a “must still work” scorecard sitting at 100% isn’t pointless, but it also isn’t telling you the AI is getting better, only that it hasn’t visibly broken yet. Those are different pieces of information, and the source is explicit that conflating them is the mistake. Not every check needs to run constantly, cost and stakes should decide the schedule Running every test on every single change is possible but expensive. The fix is tiering: cheap, simple, yes-or-no checks run on every change, because they cost almost nothing. Slow, expensive checks that need the AI to actually attempt a full task against a real system run overnight or just before a release. Tests that always pass no matter what get made harder or retired, because an always-passing test is paying money for zero information. And tests that fail for unrelated, random reasons flaky tests lose people’s trust fast; Google found that once a test’s random-failure rate crept toward just 1%, people stopped believing its results and started ignoring them, and on bad days more than 50 changes slipped past ignored test failures. The dollar amounts matter too: one full run of a well-known AI coding benchmark, at a modest per-task cost, could run into the thousands of dollars. That’s expensive enough that these runs rarely happen more than once, which is also why AI test results are rarely reported with any sense of how much they’d vary from one run to the next. When something slips through, turn it into a permanent test, but don’t expect that to predict the next new failure One illustrative incident: the AI reported that a database change’s tests passed, but the “undo” step for that change had never been written, and no existing test had ever checked for an undo step, so the AI’s self-report was technically accurate. No test written in advance would have caught it. A real study of one company’s AI system over eight weeks found a precise version of this limit: looking back at 15 incidents, the team’s safety checks had caught exactly none of them ahead of time, but once added as permanent tests after the fact, caught 87% of repeats. In 12 of the 15, the root cause was something nobody had thought to check for at all. Most silent failures were ultimately caught by a person actually reading the output, not by any automated test. This is one company’s data, so the exact percentages won’t hold everywhere, but the shape of the finding is the real point, and the author sums it up as: these checks are built to catch failures you’ve already had, not to predict new ones. That’s why the recommendation is to keep a human regularly looking at real output, on top of running tests. The practical version: decide in advance what counts as an incident worth turning into a permanent test something user-visible that shipped broken, any kind of data loss, a human having to manually intervene or revert, a task that ran way over its time or budget, or the AI claiming success when it wasn’t true . Then for each one: reproduce it, shrink it to the smallest version that still shows the problem, add it as a permanent check, note which incident it came from, and confirm the check actually fails on the old broken version and passes once it’s fixed. The test suite itself needs an owner, or it quietly rots In one illustrative example, a test started occasionally failing for an unrelated reason a database connection timing out , and whoever fixed that flakiness also deleted the exact part of the check that would have caught the real earlier bug. The fix was approved as a routine flake fix. Nobody who actually understood what that check was protecting saw the change, because nobody specifically owned it. The fix is splitting ownership on purpose: whoever runs the testing infrastructure owns the plumbing, but whoever owns a piece of real behavior, like database migrations, owns the test for that behavior and decides what “correct” means inside it. And every change to the tests themselves should go through the same kind of review as a change to a database’s structure: proposed and reviewed openly, never edited quietly. Even well-funded measurement efforts drift over time as cases get added, removed, and updated, which is a reminder that a test suite isn’t a one-time setup, it’s something that needs ongoing care like a living document. Build it early, while the requirements are still fresh Early on, what a feature is supposed to do is written down somewhere recent, so writing a test for it takes an afternoon. Wait six months, and writing the same test means digging through old changes to work out what the AI was ever actually supposed to do. One real team started late, after their AI assistant was already widely used, and still managed to build a working test system, it just took about a quarter of work. So later is harder, not impossible, and the source is careful that this is about difficulty rising, not some fixed law that costs always compound; some teams reasonably wait to build this until they’re at a bigger scale. Other guidance is stricter, naming “wait until after you ship to add any tests at all” as a mistake to avoid outright. What this means for you If you’re responsible for a coding AI or anything similar in production, approving each change as it comes in is necessary but not sufficient. You also need a separate, standing set of tests that runs on its own schedule, checks the whole system not just the latest diff , keeps “is it still doing what it used to do” separate from “is it getting better at new things,” is priced and scheduled by how expensive and how risky each test is, grows from real incidents rather than only from guesses made in advance, and has a specific person responsible for each part of it. Build that system early, because it only gets harder to build the longer you wait. The technical terms, in plain words - diff / diff review = reviewing one specific proposed change before it’s accepted, like proofreading one memo rather than auditing the whole department. - coding agent = an AI system that writes and modifies code on its own, not just suggests a line. - regression suite / regression eval = a set of tests checking that things the system already could do still work; should stay near-perfect. - capability suite / capability eval = a set of tests checking new, still-developing abilities; expected to start weak and improve over time. - pass rate = the percentage of tests that succeeded. - CLAUDE.md = a plain-text instructions file the AI reads to learn a team’s preferences and rules. - harness = the surrounding toolset and setup the AI runs inside which tools it can call, how it’s wired up , separate from the AI model itself. - model alias / snapshot = a label like “latest” that points to a specific version of an AI model; the label can silently start pointing to a newer version. - CI/CD = automated systems that run checks every time code changes, before and after it’s merged. - path-filtered check = a test that only runs automatically when specific files change. - nightly cron = a check that runs automatically every night on a fixed schedule, regardless of whether anything changed. - saturated eval = a test that always passes and so has stopped providing useful information. - flaky test = a test that sometimes fails for reasons unrelated to any real problem, eroding trust in its results. - trials = how many times a test is repeated, used to tell a real failure from random noise. - intake = the process of turning a real incident or failure into a new permanent test. - postmortem / incident = a formal review of something that went wrong, done to learn from it and prevent repeats. - suite version = a version number for the whole set of tests, so results from different points in time can be compared fairly. - LLM-as-judge = using one AI to grade the output of another AI, instead of a fixed yes/no rule. - self-report = the AI’s own claim about whether it succeeded, which can be technically true while still missing the real problem. - golden baseline / golden set = the reference answer or expected result a test compares the AI’s output against. Keep reading: the full version, with the research and sources ↓ essay Every PR to your coding agent last quarter passed review, and none of those reviews could have caught what Chen, Zaharia, and Zou measured. Under the same model name, GPT-4’s directly executable code rate fell from 52% to 10% between March and June 2023, with no diff to review Chen, Zaharia, Zou: How is ChatGPT’s behavior changing over time? https://arxiv.org/abs/2307.09009