What a second model cannot catch

A developer argues that while AI can automate much of software engineering, the irreplaceable human contribution lies in intent and judgment, not verification. The developer describes how AI agents caught each other's errors, but only a human could distinguish intended behavior from defects and make product decisions.

A lawyer told me his job was safe, and gave a better reason than most people give. He catches AI mistakes constantly, he said, but that is not the point. The point is where his work lives. Roughly 20% of it happens in courtrooms and in private negotiation — rooms that produce no transcript anyone will ever train on. The rest, the drafting and the research, is already in the corpus. His conclusion: "AI at best can rise to the level of what a law student knows after graduating." I answered honestly, and the answer was worse for me than for him: Your 20% is protected because it never gets written down. My work is 100% digital — all the code, all the solutions, everything is basically the training data. Maybe my whole job is that 80%. That is the uncomfortable version of the question. Not "will AI replace developers", which is exhausted, but: if none of my work is protected by scarcity, what is left that is actually mine? The obvious answer — orchestration, verification, taste, being the interface between the human and the machine — is also the answer everyone gives, which is a good reason to distrust it. So I spent a stretch of work watching for it specifically. Not asking whether steering matters. Asking what it catches that nothing else does. The result splits cleanly in two, and the split is more useful than the slogan. A large share of the "only a human can" list is not human work at all. It is verification work, and a second model does it well — often better than the first one did, because it has no stake in the first answer being right. I don't have to be in the room for this part, and mostly I'm not. Over one two-day stretch my agents caught each other repeatedly: a reviewer refuted a diagnosis another agent had already written up, an executor stopped dead on a brief whose premise was wrong and explained why, a fresh pass turned up ten defects in code a previous pass had promised to check and quietly skipped — including one that charges a user a fee and then denies the request. Not one of those needed me. My contribution was structural, not intellectual: a setup where a second opinion is cheap, and rules that make refusal legal — an agent that hits a false premise is allowed to stop rather than build on it. That is real, and it is not nothing. But it is also automatable , and anyone reading this can copy it in an afternoon. If this were the whole answer, the lawyer would be right about me. Then there is the other category, and it behaves differently. Case one. A feature had been built, verified, and shipped. Six independent verification passes came back green. Every one of them confirmed the code did what the spec said, and none of them was sloppy — the spec wasn't wrong either. Then I used the product, and the feature was silent. A single line on the client discarded the value the server had just computed, so the whole thing ran end to end and produced nothing a person could hear. The lesson isn't "human testing matters", which everyone already agrees with and nobody schedules. It's narrower and less comfortable: every one of those passes was asking whether the code matched the description. Not Case two. My agents handed me a list of nine defects queued for fixing. I removed three of them on sight — not because the analysis was wrong, but because the behaviour was intended . A toast notification covering the navigation bar is a defect if you assume it shouldn't; it's a design decision if I decided it should. No verification pass can separate those two, because the difference isn't in the code. It's in what somebody wanted, and I'm the one who wanted it. Case three. The plan on my desk was to build a new empty-state component for a slot that was stuck showing an infinite loading animation. I stopped it: the component already had a paused state, because I had asked for one when the design system was built. The fix went from a new component to a single boolean. There is a pattern, and it is not "humans are wiser". In all three cases the machine had every fact and still could not get there, because the missing input was not a fact . It was intent — and intent lives in the person who formed it, not in the artifact. A verifier can prove a system does what it says. It cannot tell you whether what it says is what you meant. That gap does not close with a better model, because it isn't a knowledge gap. Any version of this argument that only lists wins is an advertisement, so here is the stretch where I lost — and the interesting part is exactly where I lost it. I'm a frontend developer. The work in question was backend: a caching layer I can read but could not have designed, in a language I don't write daily. And that is precisely where I loosened up. Not on how much work I handed over — on how much judgement . In my own domain I argue with the machine constantly, because I can feel when an answer is shaped wrong. Outside it I have no such feel, so I let conclusions through. The conclusion that got through: a cache would serve stale data. It was a good-looking claim. The cache was real, its key was genuinely missing the field it was said to be missing — and the whole thing sat on a code path the application never takes. Ten call sites, not one of them reaching it. A single search settles it. For two days nobody ran the search, and I wasn't equipped to know one was needed. The cost wasn't a bug. It was a plan: work split across two repositories, one of them another team's, and a dependency communicated that did not exist. It did get caught, and how it got caught is the part worth keeping. Not by me spotting the flaw — I could not have. By me refusing to sign off. I kept pushing the plan back and forth, asking for it to be exact, making the pieces line up, and the contradiction eventually surfaced under that pressure. All of the errors came out in the end, but not one of them came out because I knew better. So the thing the seat supplies isn't expertise — I had none here, and it still worked. It's the refusal to accept "close enough" from a system that is fluent, fast, and confident. That is a much smaller qualification than domain knowledge, and much easier to give up, because giving it up costs nothing today and feels like trust. Back to the lawyer. His 20% is protected by scarcity: those rooms produce no data. That protection is real but passive, and it only holds while the rooms stay closed. The seat I'm describing has a different property. It isn't protected by missing data — it's the seat that supplies something data cannot contain, which is what "correct" was supposed to mean in the first place. No corpus confers that, because it was never a fact about the world; it was a decision somebody made. But I want to be precise about the size, because this is where these arguments usually inflate. That seat is not 80% of my job. Most of my week is still specification, review, and repair — all of it in the corpus, all of it increasingly done better and faster by the machine. The part that is structurally mine is small, intermittent, and mostly consists of knowing what I actually wanted. Which leaves the real risk, and it isn't replacement. My wife doesn't work in tech. She read an earlier version of this and gave me the sharper version of it: I changed my mind a bit about AI and robots. It's not going to be some uprising because they don't need us. Humanity will be at fault, because it'll stop controlling and checking, and it'll just flow that way on its own. And she named the cause without softening it: Because of human stupidity and degradation, in a way. Like — whatever, why check, it does it itself. What happened next is the reason this section exists at all. I put that to one of my agents, and it returned a clean economic account: checking is boring and expensive, the machine is right most of the time, the cost of verifying exceeds the expected value, so people rationally stop. Tidy, plausible, and wrong in a way that is hard to see, because it is the kind of wrong that reads as insight. There is no economic argument. "I don't care" is not a weighed cost — the weighing never happens. That is exactly what makes it durable: you cannot refute a decision that was never made. Give it a model and you have already mistranslated it into something answerable. The operational version isn't "checking is too expensive". I can write it out myself, because it is already how I think on a good week, when everything is landing: I just won't check, because it all works anyway. Why bother doing extra work if everything's fine. When the mechanism visibly starts failing, then sure, you can check. But by then it's already too late. So the seat doesn't disappear because something takes it. It disappears because sitting in it stops feeling necessary — and the signal that it was necessary arrives after the fact, if it arrives at all. Her conclusion was one sentence, and I have not improved on it: you can't let it run by itself, you have to keep some control and know what's going on. The lawyer exchange is from a Reddit thread. The framing about training data, and the closing line — that "mashing keys" disappears while "being the interface between the human and the machine" remains — are his.