The Expensive Part Is Knowing Whether the Code Is Any Good

A CHI 2026 controlled study by Fan et al. found that AI coding assistants cut perceived workload by 18.2 points on the RAW-TLX scale, reduced task completion time 22% (25.0 vs 32.1 minutes) and raised correctness (odds ratio 1.71), but that fatigue rose more steeply in the AI cohort over repeated use, with a constructed "verification load" index mediating roughly 26% of the fatigue effect and 24% of the stress effect. A second study, PatchTrack by Ogenrwot and Businge, tracked 645 ChatGPT-generated snippets across 338 pull requests in 255 GitHub repositories and found a median full-integration rate of just 25%, with the rest selectively edited, restructured or dropped.

Part 4 of... several articles on the impact of AI on the SDLC The new asymmetry The agent finishes, the diff appears, it looks completely plausible - sensible variable names, a test file that runs green, a commit message that reads like a person wrote it - and you sit there for ages trying to work out whether you believe it. Not whether it compiles. Whether you believe it. That gap, between the diff turning up and you actually believing it, is where the cost has moved to. Writing the thing has got cheap, but deciding whether it deserves to exist, whether it’s the right solution at all rather than just a solution, hasn’t got cheaper at anything like the same rate. The output arriving quickly does not make the decision to trust it arrive quickly, and I think that mismatch is turning into the actual bottleneck in a lot of teams, even though nobody’s job title says “verification” on it. What even counts as verification, anyway People hear that word and picture someone running pytest and shrugging, and it’s a lot bigger than that, it’s really two different jobs, and AI treats them completely differently. There’s mechanical verification - does it compile, do the tests pass, does the linter complain - which is basically automatable and getting more so by the month. And there’s contextual judgment - does this fit what we’re actually trying to do, is this an acceptable risk, is this the right abstraction, will this bite us later - which nothing I’ve read suggests is getting cheaper at all. Most “AI is creating a testing bottleneck” takes conflate the two. Somebody actually measured the fatigue A CHI 2026 study by Fan et al., When Help Hurts: Verification Load and Fatigue with AI Coding Assistants https://doi.org/10.1145/3772318.3791176 , is the first thing I’ve found that actually puts a number on this. It’s a proper controlled experiment - 60 participants, three Python tasks each, split across inline AI suggestions, a chat-based assistant, a more structured AI mode, and a no-AI control, with the backend held fixed across the AI conditions. And AI helped, plainly. Workload dropped by 18.2 points on the RAW-TLX scale, a validated measure of how taxing a task felt, task completion time fell by 22% 25.0 minutes against 32.1 , and correctness went up, with an odds ratio of 1.71 in favour of the AI conditions. So this is not a “well actually AI makes everything worse” study. What they also built was a “verification load” index - compile and test failures, how long it took to get to a first successful compile, how much churn there was in the code, how many pauses and context switches happened along the way - and over repeated use, fatigue rose more steeply in the AI cohort than in the control. Verification load didn’t account for all of that, but it mediated around 26% of the fatigue effect and 24% of the stress effect. Assistance and burden coexist in the same task, and the burden is specifically the checking. AI gives you back time on the doing and quietly hands some of it straight back on the checking, and because the checking doesn’t look like “work” the way typing code does, it’s very easy to undercount. Only a quarter of it actually survives The second study is PatchTrack https://link.springer.com/article/10.1007/s10664-026-10869-5 , by Ogenrwot and Businge, which looked at 338 real pull requests across 255 GitHub repositories where developers had openly admitted to using ChatGPT, tracking 645 ChatGPT-generated snippets against 3,486 actual developer patches to see what happened to the AI’s suggestions on the way to being merged. The headline number is that the median rate of full integration - the suggestion going in essentially as offered - was 25%. The rest got selectively picked apart, restructured, refined over several passes, or dropped entirely, for reasons that were mostly mundane: it didn’t match the scope of the change, it would’ve been harder to maintain, or it duplicated something that already existed. Bear in mind these are people who admitted to pasting stuff out of ChatGPT, which is a pretty different crowd to someone running an autonomous coding agent today, so it’s a quarter of that , not a quarter of everything. Even so, it’s a useful reframe. The AI’s output here wasn’t functioning as a finished implementation, it was functioning as raw material for someone else’s reasoning - something to extract from, argue with and shape - which is much closer to how most people I know actually use these tools than “AI writes the code now”. It’s the same idea as the “candidate implementation” from the first post, and PatchTrack is a rare look at what the evaluation of a candidate actually involves. The job turns into supervision Annie Vella and Kelly Blincoe ran a longitudinal study https://arxiv.org/abs/2605.23135 that ties the first two together. They surveyed developers across 28 countries between October 2024 and April 2025 - 158 eligible participants in the first round, 101 in the second, 95 matched across both - and asked what had actually changed in how they worked, not just whether they liked the tools. 82% reported spending less time writing code. What filled the gap was directing the AI, evaluating what it produced, and correcting it, which Vella describes as a shift towards “supervisory engineering work”. Perceived productivity stayed high throughout. And then, among the matched participants, the proportion reporting a deterioration in at least one developer-experience dimension - things like cognitive load and flow, the sense of actually being in the work rather than watching it happen - rose from 14% to 27%. People aren’t doing less work, but the shape of the work has changed from producing to supervising, and supervising a pile of other systems’ output is a different cognitive job altogether to doing the thing yourself, one a decent chunk of people find harder to stay engaged with even while they report feeling more productive. It’s people reporting on themselves over time rather than a controlled experiment, so I’d believe that things got harder for a fair few of them a lot more than I’d believe it was exactly 27%. This isn’t a new problem, there’s just more of it now What gets missed in basically every “AI is breaking code review” take I’ve seen is that code review never scaled cleanly with the size of the change, long before any of this. Jureczko, Kajda and Górecki https://doi.org/10.1049/iet-sen.2020.0134 ran three separate experiments inside one company and found review effectiveness drops faster than you’d expect as the change grows, accelerating rather than in a straight line, with anything over about ten files landing in properly poor territory. A separate controlled experiment https://pmc.ncbi.nlm.nih.gov/articles/PMC7924728/ found that deliberately decomposing a change into smaller pieces reduced the number of issues reviewers wrongly flagged and changed how they went looking for context, though it didn’t improve every outcome they measured. And a study https://link.springer.com/article/10.1007/s10664-022-10205-7 that classified 1,510 individual changes made during review across Android, JGit and a Java client project found that a bigger initial patch, and more lines added up front, reliably predicted more changes happening during the review itself. None of that is about AI, it’s just how human attention behaves against a diff. What’s changed is the input. If AI-assisted changes tend to be bigger https://theotherjenai.substack.com/p/more-not-faster , then you’re feeding a mechanism that was never linear in the first place exactly the kind of input it already handled worst. Hang on though, can’t AI check its own work too? I don’t want to leave this as just “AI generates code faster than humans can possibly test it”, because I don’t think it’s true. Mechanical verification is exactly the kind of thing AI is also good at - generating unit tests, running them, iterating, static analysis, fuzzing, checking types and contracts, even having a go at repairing its own failures. There’s recent research on LLM-generated unit tests https://doi.org/10.1016/j.jss.2026.112834 showing strong coverage and mutation scores in bounded settings, alongside some fairly predictable blind spots around edge cases like None , infinity and NaN , and I’ll dig into that properly in the next post. For now the point is that “AI checked it” and “AI checked the weird edge cases a human would’ve thought of” are not quite the same claim. So a decent amount of the testing can itself get cheaper, and mechanical verification capacity rises roughly alongside generation capacity. What’s left over - the bit that doesn’t scale - is judging whether the checks that got run were even checking the right things. So what’s actually left that’s scarce Then there’s the stuff no amount of automated checking makes go away on its own. Did we actually understand the requirement, or a plausible-sounding paraphrase of it? Is this the right abstraction, or just a working one? Is the risk on this migration actually acceptable? Are these tests testing the requirement, or just agreeing with whatever assumptions the code generation already baked in? Is this safe in production, with the actual data, under actual load? Does it create an operational burden nobody notices until it pages someone at 2am? Will the next engineer who opens this file be able to tell what it’s doing and why? And then the two that never had an automatable answer to begin with - should we merge this, and honestly, should we have built it at all. That’s judgement scarcity rather than a verification bottleneck, and the distinction matters. A bottleneck implies you need more throughput, more reviewers, more coverage, more automation, and all of those help a bit. Judgement scarcity means the constraint isn’t capacity, it’s knowledge that lives in people’s heads about this particular system, this particular customer, this particular set of trade-offs, and that doesn’t get manufactured faster no matter how good the tooling gets. What if the diff came with receipts Right now a reviewer gets a diff and has to reconstruct the intent, what it touches and what assumptions it’s making purely by reading code and guessing. What if a change arrived with its own evidence attached: what it was meant to do, which contracts it affects, which checks were actually run and what they found, the known risks, and how you’d roll it back. Not more code to read, a different kind of artefact to assess, so attention goes to the consequential claims rather than re-deriving basic facts. It’s not a worked-out answer, more of a “what if”. Which brings us back to the diff on your screen But that PR’s still sitting in your queue, green tests and all, waiting for someone to decide whether they believe it. That bit was always the expensive part, it’s just that there’s an awful lot more of them now, and not an awful lot more of us. Which does leave the slightly annoying question of whether all this abundance is making the code itself any worse, and the honest answer to that one is a lot messier than either the doom-mongers or the hype merchants would like. Other articles in this series: