{"slug": "tech-history-how-does-the-human-get-out-of-the-loop", "title": "Tech History: How does the Human get out of the Loop?", "summary": "A tech history analysis reveals that the psychological challenge of moving from human-in-the-loop to human-on-the-loop in AI coding mirrors earlier transitions in aviation, nuclear plants, and weapons systems. The author identifies four established criteria from those domains that apply to AI coding, plus one new criterion unique to AI, arguing that the shift is 90% psychological and 10% infrastructural.", "body_md": "# Tech History: How does the Human get out of the Loop?\n\n### Every mature automated technology already has real criteria for human-in vs. human-on-the-loop. AI coding added exactly one new one. How does the Human get out of the Loop?\n\n## Trigger: getting out of the loop feels psychologically hard\n\nIn my courses [“AI Augmented Product Management” ](https://ueberproduct.de/seminar/ai-augmented-product-management/)- in which we go deep into how Product management looks like under full exploitation of the new tools like Claude Code / Codex etc. - we talk about the Shapiro Scale. The interesting levels are 2 and three, because getting from 2 to 3 means a tough psychological step of a) trusting the tools and b) letting go of knowledge that - until now - defined you.\n\n**Level 2:** reading every code diff line by line before it ships.**Level 3:** trusting the test suite and only checking whether the feature actually works.\n\nI didn’t think about it as and called it “**human-in-the-loop versus human-on-the-loop**” **(HITL vs HOTL)** until I wrote [the last article on here,](https://intentful.ueberproduct.de/p/human-in-the-loop-required-or-was) and then it was obvious that’s exactly what it is. And, the discussion of “the human needs to stay in the loop” is getting full coverage, sadly most of the discussion is orally laden, with little facts. That’s what I touched on in the last article: is there a huge difference between automating red lights and coding?\n\nThe rulebook don’t exist, which give you an answer for when you’re allowed to move from one to the other. No one tells you, when it’s ok, to leave the top and trust the agents to get called in in escalation mode only. Watching it happen in my training cohorts now and seeing the doubt, my estimate is something like a thousand repetitions before it feels safe. The move is roughly 90% psychological, and 10% infrastructure (even though that’s also a tough engineering task). The ratio still feels strange. Ninety percent *feeling* is an awkward way to get to what should be an engineering decision. My guess was: this is not the matrix, not teh singularity. It must have happened before. So I went looking for whoever had already answered this properly.\n\n**Tech history repeats itself **\n\nTurns out plenty of people had. Aviation, nuclear plants, weapons systems and self-driving cars all had to deal with this exact question decades before even the idea of a coding agent existed: when does a human check every step, and when does a human just watch and step in if something looks off? The reasoning in the past was a bit less biased on *vibes or moral verdicts*. They wrote criteria down, argued about them, and **put them into standards documents** you can still read.\n\nFour of those criteria apply to us directly. AI coding added exactly one that wasn’t available to the older technologies. And then, finally, there’s a fifth one that gets no mention. I’ll handle that last.\n\n**Aspect 1: Can a person even react in time?**\n\nThis is the oldest and hardest one, and the only one that can be decided with a stopwatch in your hand.\n\nThe US Navy’s Phalanx close-in weapon system, deployed since 1980, normally runs under a human’s direct control - until the number of incoming threats outpaces the capability of a person to track them. At that point the operator flips the system to automatic. Yes, it runs on manual as long as things are simple. As soon as they become complicated and overwhelming, the human trusts the loop more than himself. Interestingly, nobody argues about this one on principle. Incoming munitions don’t wait for a committee to decide correctly and in full alignment.\n\nCars do the same thing and there is only little discussion, it simply gets implemented in the background: automatic emergency braking never went through a human-in-the-loop phase at all. It couldn’t. The reaction window is a few hundred milliseconds, well under what a person can manage, so the standard that defines self-driving levels ignores it completely - it’s not even on the ladder from Level 0 to Level 5, because it was never a candidate for human oversight in the first place. You’ve probably felt this yourself: the car braking half a second before you even could. Nobody wrote a post demanding a human stay in that loop. (Although experiencing phantom breaking by early Teslas might make you appreciate the “[Shopping Cart Theory”.](https://en.wikipedia.org/wiki/Shopping_cart_theory))\n\nNuclear reactors run on the same principle. Under certain fault conditions the control rods drop automatically, because - in the words of the researchers who formalised this back in 2000 - **“the operator cannot reliably respond in time to avoid an accident.”** Let that sink in! The level of trust! If a human genuinely cannot act inside the window, keeping them in the loop is theatre. and in those existential situations, in that tech, we simply switch to automatic. End of discussion.\n\nAs another example, there’s MCAS. Boeing’s system on the 737 MAX assumed pilots would recognise an unexpected activation and correct it within three seconds. The FAA approved that number. Two crashes and 346 deaths later, it turned out pilots couldn’t do it reliably - not because they were bad at their jobs, but because the number was wrong and nobody had tested whether it was right. The criterion made sense. The number didn’t, and nothing warns you about that until the thing you relied on top of it fails.\n\n**Aspect 2: What does the mistake actually cost?**\n\nReaction time tells you whether a human can act. The second criterion asks what happens if nobody does, or if they act wrong.\n\nCivil aviation avionics are held to roughly a one-in-a-billion catastrophic failure rate. The number comes from formally multiplying **how likely a failure is by how bad it would be.** I go through the same calculation, far less formally, every time I decide whether an agent gets to touch a client deliverable or just my own side project. Low cost, low probability: let it run. High cost, any probability: stay close.\n\nYou’d expect that logic to show up explicitly in the actual standards. Mostl of them don’t show them, which is worth knowing before you think of them as settled. SAE J3016, the document that defines self-driving levels, doesn’t grade cars by risk at all. It grades them by who’s doing the driving and who catches it when something goes wrong - a raw role assignment, not a risk score in sight. Phil Koopman, who wrote the standard’s own field guide at Carnegie Mellon, is straight about the myth that was cerated around them: “higher level numbers have more automation, but they might not be safer.” Level 3 is simply a different arrangement of who’s responsible for what. The higher number doesn’t indicate less risk or danger.\n\nThe US military does the same thing on purpose. Its directive on autonomous weapons defines no explicit risk threshold at all that flips a system from supervised to autonomous. Michael Horowitz, who helped write the 2023 update, says **the omission was deliberate:** the standard is **“appropriate levels of human judgment,”** a phrase built to *stay flexible rather than to compute*. So if you want a clean formula for how much risk buys how much autonomy - the people who do this for a living, with actual lives on the line, decided not to give you one. That seems oddly comforting to me. It seems we currently aren’t failing at something everybody else already solved.\n\n**Aspect 3: The one new criterion - can the machine check itself?**\n\nNone of the older technologies had this one available, because it didn’t exist: the artifact checking itself.\n\nBoris Cherny, who built Claude Code at Anthropic, described his own working style in February this year as still needing a human eye on the output - “you still do want a human looking at the code.” A month later Anthropic shipped a feature where a team of agents reviews every pull request instead, and Cherny’s explanation was refreshingly unromantic: reviews had become the bottleneck. In other words, economics made him look at and try to solve the new bottleneck. Whatever. By June he was saying he hadn’t hand-written a line of code in eight months, and that most mornings he wakes up to pull requests Claude already built, tested end to end, with screenshots proving the feature runs.\n\nFour months, start to finish. What moved wasn’t trust arriving on a schedule - what happened was that the checking moved from a person into the artifact itself. Code can run its own tests. That’s the whole trick. The trick is that you have loops that interact with each other: produce - check -produce - check until a halting criterion is fulfilled.\n\nA PRD is an artefact that does not have that capability. It can’t run itself and tell you it’s wrong, and that makes me very uncomfortable about my own profession: we product (well, and “business” in general) people have never cared about “correctness”. Developer agents know when to stop because linters, type checks and tests hand them a closed loop. We have no equivalent for a PRD, a strategy deck or a hiring decision - partly because it’s genuinely hard, partly because we found it more flattering to call the gap “judgment” and leave it at that. And we repeat the pattern now by a ton of “product leaders” calling for “taste” and judgement as the last line of defence for our jobs. I call that BS and simply privilege.\n\nVerifiability is the newest criterion on this list, and it’s the one where the rest of knowledge work is sixty years behind software. You can value the compiler and what followed high enough. The insight that going is math and ctagroy theory and completeness and that teh question if P (N(P) is important makes all the difference. The rest of knowledge work takes a bath in ignorance and subjectivity, but “data driven”, please. Make of that what you will. Maybe it’s an art form.\n\nThe problem: It’s also the easiest one to fake. Cat Wu, who works alongside Cherny, named the failure mode directly: if a team waves through 99% of an agent’s permission prompts, the review step exists on paper and does nothing in practice. The military realised this decades ago and wrote it down - with C-RAM systems the operator holds a veto they have half a second to make use of, and, as one analyst put it, “**few [are] willing to challenge what they view as the better judgment of the machine.”** In other words, pragmatically speaking, they trust the machine more than they trust themselves, if asked. Same ceremony, different uniform. And - give that a minute to sink in - this is the closest thing to a controlled study we have: sixteen experienced developers, using Cursor, working in their own real repositories found them 19% slower. **Verifiability being possible in principle says nothing about whether it’s running in your pipeline.**\n\n**Aspect 4: Do you actually still know how?**\n\nNone of the official standards mention this one, and a regular in my AI-augmented PM cohort put it better than any paper I’ve read: “Breakdowns happen exactly at the edge of one’s own competence, not inside it.” Where you’re genuinely good, you can hand more to an agent, because you’ll spot the moment it drifts. Where you’re not, it fills the gap plausibly and with full conviction but you you have no way to tell. Plausible is the dangerous part. The output always looks smooth, no matter if it’s right or wrong.\n\nOne might think that settles it - get skilled first, then extend autonomy. Except the skill decays exactly where you need it. Lisanne Bainbridge wrote this down in 1983, long before any of this technology existed: automation hands the routine work to the machine, and the human left watching never gets to practise the skill they’re supposed to use when something breaks. This sounds like a terrible fate. But there is an obvious solution. The FAA eventually had to write a memo about it. Pilots who’d stopped hand-flying because the autopilot did it better were losing the ability to fly the plane when the autopilot quit - so in 2013 **the agency told airlines to mandate manual flying hours **as a counter measure. On purpose. To keep alive a skill that automation had made unnecessary, until one day, in urgency, it wasn’t.\n\nThe Level 3 Product Managers in my courses didn’t get there by getting smarter about “the code” (or outcome as they are PMs). Most of them read less of it than the year before. The ones who can still jump in when something breaks are the ones who deliberately still messed with it..\n\n**The real driver behind automation**\n\nEvery criterion above sounds principled: measured risk, measured reaction time, measured verifiability. Then you look at the actual history and research - and the boring answer driving automation is always the same: Money, $$$, Mullah, the economy.\n\nNew York City automated its traffic lights in the 1920s, and the reason on record wasn’t safety data. The city could reassign all but 500 of its 6,000 traffic officers and save $12.5 million doing it. (Guess what the officers thought about the reliability of the new technology.) The US Air Force’s own 2009 planning document, arguing that operators should move from “in the loop” to “on the loop,” made its case with a staffing table: the same number of patrols that needed 570 pilots would need 150 once multi-aircraft control kicked in. Trust was not mentioned in the document as a concept. As long as the miss / hit rate made the program cheaper, everything was fine.\n\nProbably the same is true for a decent share of the AI-agent autonomy pitches landing in your inbox right now. When someone tells you their tool has earned more autonomy, ask which calculation actually changed - the risk math, or the headcount math. To be frank, it’s usually the second one wearing the clothes of the first. Especially in an industry that is terrible at calculating development productivity or - even worse - product outcome. Ask Kohavi who made a career of trying to understand that and came top with the numbers.\n\nBy intuition, we basically gamble as Product managers: 1/3 of what we realise has positive impact, a third has no measurable impact, yet another third has bad influence on value related for the customer. In other words: The bar is as low as throwing the dice.\n\n**Finally**\n\nI still ask my course participants where they are on the Shapiro Scale, and I still watch most of them walk up the ladder quite slowly and trying to “stay in control”. The instinct is to look into the agents, to control at the wrong, tedious level. Until there is trust. The reason is not that nobody gives them the criteria - that’s one of my first moves. It’s because **knowing the criteria and trusting them are two very different jobs**. The technologies that solved this before us didn’t simply skip those levels. They had a few more decades to consider than we’re getting. We didn’t choose for AI to hit this hard and fast. But here it is and the best thing is to try and understand it and look for history examples on how to deal with it.", "url": "https://wpnews.pro/news/tech-history-how-does-the-human-get-out-of-the-loop", "canonical_source": "https://intentful.ueberproduct.de/p/tech-history-how-does-the-human-get", "published_at": "2026-07-18 14:25:47+00:00", "updated_at": "2026-07-18 14:26:15.274994+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-policy", "ai-agents", "developer-tools"], "entities": ["Claude Code", "Codex", "US Navy", "Phalanx"], "alternates": {"html": "https://wpnews.pro/news/tech-history-how-does-the-human-get-out-of-the-loop", "markdown": "https://wpnews.pro/news/tech-history-how-does-the-human-get-out-of-the-loop.md", "text": "https://wpnews.pro/news/tech-history-how-does-the-human-get-out-of-the-loop.txt", "jsonld": "https://wpnews.pro/news/tech-history-how-does-the-human-get-out-of-the-loop.jsonld"}}