AI Harness Engineering · Essay Five · derek wang (derekwang85)
Deming said something that makes factory owners uncomfortable: quality cannot be inspected into a product. You build quality into the process, not into a final check. He was right — but in the AI era there's a side to it everyone skips. The quality you build into the process still has to be checked to confirm it actually got built in. Deming was against inspection as the only source of quality. He was never against having a check.
In an AI codebase, the check matters more than it ever has. Everything before this essay is a constraint: strategy, architecture, contracts. Files that say how things should be. But a file doesn't execute itself. An AI will read it, and it will also quietly drift away from it under pressure and ambiguity. Between "the constraint is written" and "the code actually did that" sits an unguarded canyon. The gate is the bridge — the check that comes before the code is accepted.
Traditional engineering has gates. CI runs compiles, static analysis, tests. Every mature team has them. But every one of those gates shares an assumption — that the code was written by a human.
Human code has human fingerprints: it forgets, it gets lazy, but it doesn't confidently fabricate. AI code is a different animal entirely. Let me put the numbers on it, all from public reporting:
An analysis of pull requests across 470 GitHub repos found that roughly 1 in 5 AI code samples referenced a library or API that doesn't exist — not a typo, a straight-faced invention [Report: CodeRabbit analysis of AI code hallucination, 2026 Q1].
Google's research reported a stark split: AI-generated code hit about 92% syntactic validity, but only 78% logical correctness, and fell to 54% on complex scenarios like concurrency, transactions, and exception branches [Research study: Google on LLM code-generation correctness, 2025].
The Cloud Security Alliance put the share of AI-generated code that fails basic security scans between 45% and 70% [ORIGINAL DATA], while 75% [ORIGINAL DATA] of developers in the same window believed AI-written code was safer than their own [Report: CSA on AI coding security, 2026-04].
Read that closely: the syntax is almost never the problem, so it passes every old gate. The wrongness lives in the semantics. Linting, compiling, and formatting all work at the syntax layer. You're using a spellchecker to catch an essay that argued the wrong point. It checks the typos faithfully and still lets the false argument through.
My own TradeOMS work hit exactly this. Early CI was green, all tests passed, and AI-sneaked logic errors still made it into the trunk: a null assertion written as a tautology, an exception swallowed into a "success," == typed as =. Syntax gates all passed. Every semantic error sailed in.
Before the mechanism, the distinction, because it holds the whole system together:
A constraint is text — the file, the spec, the convention. It declares should.
A gate is execution — the script, the test, the rule. It verifies is.
A constraint without a gate is a promise nobody checks. A gate without a constraint is a check nobody asked for.
Essay One gave the pyramid: strategy → architecture → contract → gate layer → implementation. Now the gate layer earns its place: it's the closure. Without it, the four layers above it are paper.
The physical shape in the framework is a real, verifiable structure — a dispatch/ directory holding gate scripts, the full-regression test system, the triple-track ledgers (WBS / Issue / Test Case), and the multi-role swarm for parallel review, with a tests/fulltest-baseline-R1.md as the regression baseline [ORIGINAL DATA]. The core isn't any one script. It's a discipline: an AI that changes code cannot pass its own work. It must run a gate it cannot edit. The gate is defined by a human, executed by a script, judged by an independent mechanism. That rule matters more than the individual checks, because it names the AI's real weakness: an AI has no self-doubt. It is permanently confident in its own output. Let a non-doubting executor grade its own work and the grade is always a perfect score — which is exactly what "self-review" produces.
On the coding side, five gates settled in, mirroring the writing side: G1 style, G2 structure, G3 facts, G4 consistency, G5 independent review. Each one blocks a specific failure: AI-flavor style, docs drifting from constraints, invented APIs and invalid assertions, "fix A forgot B" ripples, and self-grading.
Five gates are a pre-release line. But gates themselves age: rule libraries go stale, the AI finds new workarounds, baselines drift. So the codified version is a six-stage ladder that runs the whole lifecycle — G0 baseline → G1 compile → G2 three-layer analysis → G3 ripple scan → G4 re-test verify → G5 experience hardening → G6 sign-off release — moving from "block bad output" to "a gate system that gets sharper the more you use it" [ORIGINAL DATA].
Three design principles matter more than the check list itself:
Principle one: a gate must quantify. "Code quality should be high" is not a gate. "First-pass rate ≥70%" is [ORIGINAL DATA]. An unquantifiable gate can't regress, can't compare, can't improve. It's not a gate, it's a prayer.
Principle two: a gate runs before acceptance, not during writing. Lining every line of code with checks kills output; lining every sentence with checks kills expression. A gate's value is in blocking, not in coaching.
Principle three: failure needs a return path. A gate that fails is a router, not a dead end — G1 failure routes back to rewriting, G3 back to the source-material stage, G4 back to a ripple scan. A gate with no way back just produces the frustration of a stuck submission, and frustration pushes people around the gate.
The whole point of a gate, in one takeaway line: check before code, and the error is stopped before it ships instead of after — a constraint writes down how it should be, a gate proves it actually is.
The first question is always money: won't more gates slow development down?
Measured against the alternative, no. Full CI time for the five-gate suite runs in minutes; a single cross-module human review runs in hours [ORIGINAL DATA]. That's the economics in its simplest form — the check is a script's job, and a human only reviews what the script can't decide.
The other ledger is rework. CodeRabbit's 2026 Q1 study of 470 GitHub repos put AI co-authored code's bug rate at roughly 1.7× the human baseline, averaging 10.83 issues per AI segment versus 6.45 for human-written code [Report: CodeRabbit on AI bug-rate, 2026 Q1]. GitHub disclosed that about 46% of submissions from the average Copilot user is AI-generated, and 61% for Java developers [Origin: GitHub public disclosure]. The more an AI writes, the more semantic risk there is. Gate money isn't spent on "a few extra seconds." It's spent on converting hours of human triage into minutes of automatic interception.
I have to give the critic their due, because gates aren't free, and over-gating is a real failure. Rules that are too rigid flag normal AI output as violations and push the model into filling-checkbox theatrics that lower quality. Pile every gate into a single pre-commit and one submission takes fifteen minutes, so developers find a shortcut around it — and the gate system quietly stops existing. Correct every function's try-catch, enforce a norm on every comment, and the code is "clean" and pointless.
All three objections hold. So the real difficulty isn't "add how many" — it's "don't overdo it." Two countermeasures: gate tiering — only true constraints (architecture, contracts, security red-lines) get hard gates, everything else gets a soft hint, and the fewer hard gates the fewer false positives and fewer reasons to route around them; and self-growing gates — the rule library records failure modes, baselines stay adjustable, false positives can be appealed. A gate isn't a verdict; it's an instrument that gets sharper.
One sentence for the whole counter-side: no gate, and semantic errors leak into the trunk; too heavy a gate, and the product is strangled by rules. Health lives between the two — block the semantic errors, let the normal work through.
A side note: the same discipline runs in my knowledge base, derekinside. Every piece of knowledge passes an ingestion check before it's admitted — chunk format, entity-relation validity, conflict with existing knowledge. Knowledge isn't "recorded, therefore counted." It's "checked, therefore counted." That's the same law as code gates: the entrance can be loose, the exit must have a gate. A knowledge base isn't kept clean by writing carefully; it's kept clean by checking before admission — exactly as AI code isn't kept reliable by generating carefully, but by checking before merge.
Deming's line, translated for the AI era, reads like this: quality isn't inspected into a product — but quality without a check is only claimed. Writing the constraint is the first step. Making the constraint execute is the second, and the gate is the second step. It's what turns "we followed the rules" from something we say into something that's a fact.