The AI writes the tests. It doesn't get to grade them.

A developer proposes a rule for AI in testing: the producer of a test should not be its validator. The approach, implemented in an open-source pipeline, uses separate models for writing, judging, and debugging tests, with a human-readable feature file as the single source of truth.

Originally published at dmitriiev.dev Here is the rule the whole thing is built on: every stage is checked by something that did not produce its output. A parser, a different model, a real exit code, a person. No agent gets to declare its own work done. That is not an AI idea. It is one of the oldest in engineering under a newer name: the thing that makes something should not be the thing that certifies it. Compilers, CI, code review, mutation testing all run on it. Producer is not validator. Most of the 2026 wave of AI in testing forgets it. The pitch is to let an agent write and run your tests. A lot of what ships drops a model into the hot path of CI: the same kind of system that wrote an assertion also decides, on every build, whether that assertion is reasonable. That is a closed loop. A model is good at inventing a plausible test and much worse at catching its own wrong assumptions, and letting it grade itself just lets it be confidently wrong twice in the same direction. Writing tests is the cheap part an LLM is genuinely good at. Proving they are worth running is the work, and it takes someone other than the writer. So the writing here is almost incidental, and every artifact a model produces gets handed to something else to check. None of this is a whiteboard design; it comes out of maintaining real Playwright suites, not out of a diagram. The whole thing is public, demo app included, and runs end to end in a few minutes: ai-qa-pipeline https://github.com/VladyslavDmitriiev/ai-qa-pipeline . You describe a feature in plain UI language, no selectors and no code: Feature: Checkout Rejects an invalid card number - Add a product to the cart - Go to checkout and enter card number 1234 - Submit Expect: an error message says the card number is invalid Before a single model is called, a parser reads that file and can reject it for free. No feature heading, no scenarios, or a scenario without exactly one Expect: line, and it stops right there. That last one is a real constraint, not a formality: if you cannot name one outcome for a scenario, the contract makes you split it. One scenario, one assertion. It has an ergonomic cost, a checkout you think of as one journey with four things to verify becomes four scenarios repeating the same steps, and the trade is deliberate: one clean primary assertion per test is what makes the mutation check downstream mean anything. Each box hands off to a different kind of judge: feature.md │ ▼ Feature Reviewer ── is this testable, or would a writer have to guess? │ pass ▼ Writer ◄────────┐ │ draft │ reject with a problem list ▼ │ Judge a different model ── approve, or send it back │ approve ▼ real Playwright run ── green by exit code, not by a model's opinion │ red ▼ Debugger ── may fix mechanics, must not touch meaning │ green ▼ Judge again ── did the fix quietly weaken anything? │ ▼ tests/generated/ scratch, untrusted The Feature Reviewer judges testability, not taste. It rejects a scenario only when a step is not a concrete user action, or the expected outcome is not observable in the UI, or the scenarios contradict each other. It is told in as many words not to reject for style or for edge cases it would have added. Scope is the feature owner's call, not the agent's. The line it draws is concrete: Submit checkout, expect checkout to work gets rejected, because "works" is nothing the UI can show, while Submit with card number 1234, expect an error that the card is invalid passes, because there is exactly one observable outcome to assert. A rejection stops the pipeline and prints what was wrong, and the fix is editing the feature file, not arguing with the model. The writer works from ground truth, not vibes. It gets three things and nothing else: the feature file, the app's actual source, and one hand-written frozen spec as the house style. Every data-testid and every piece of UI text it uses has to exist in that source. The app source is the anti-hallucination anchor: selectors get copied from what is really there instead of imagined from what a shop probably looks like. That anchor is not free, though. Someone has to point the writer at the right source, and on a real app curating that slice is manual work, which the last section is honest about. The judge runs on a different model on purpose. Writer on Sonnet, judge on Opus by default. Be honest about how independent that actually is: both are Anthropic models trained on overlapping data, so this is a weak form of producer-is-not-validator, not a clean-room second opinion. The value is narrower and still real. The judge is a separate, more capable instance with no stake in the draft it is reading, so it does not defend the writer's habits the way the writer does. The point is the independence, not Opus; the judge is a config flag, and pointing it at a different vendor entirely buys you more of the same thing. It is not magic either way. Both read the same static source, so a selector that only breaks in the rendered DOM can slip past both, and that is what the real Playwright run downstream is for. What a second model reliably catches is the writer rationalizing its own draft: a tautological assertion, a missing await , an Expect: outcome asserted more weakly than the feature asked for. The write-and-judge loop is capped at three drafts. Three rejections kill the run with the judge's last problem list attached, and in practice that almost always means the feature file was ambiguous. If the real run comes back red, a Debug Agent gets the spec, the Playwright output, and the app source. Its contract is asymmetric, and the asymmetry is the whole point. It may fix locators, waits, navigation, ordering, test data. It may not change what any test asserts: expected text, counts, URLs, the choice of primary assertion. And if the spec is red because the assertion actually contradicts how the app behaves, that is not a bug to paper over. The agent returns NOFIX: <why and the pipeline stops with that reason, because a spec that can only go green by asserting less should fail loudly. The debugger is told not to touch assertions, and the pipeline does not trust the telling. Any spec that went through debugging goes back to the judge after it turns green. The judge never sees a diff, and it does not need one: it re-reads the primary assertion against the fixed reference it used the first time, the feature file's Expect: line. An assertion quietly loosened to force a pass, exact text swapped for a bare visibility check, no longer matches what the feature demanded, and gets rejected on a green run. Two honest limits. The re-read is a model judgment, not a proof, so it is only as strong as the Expect: line is specific: a vague expectation still buys a vague assertion. And it guards the assertion, not the whole mechanical contract, so a debugger that swallows a failure in a retry wrapper or swaps a UI step for an API shortcut is not something the assertion check alone will catch. What it does shut is the most common way agentic fixers cheat: optimizing for make-it-pass by asserting less. Every path out of this system is bounded and named. Three drafts, two repairs, one re-review. When it cannot produce a spec it can stand behind, you get a PipelineError carrying the evidence, the judge's problems or the Playwright tail or the NOFIX reason, not a weakened green. The happy path is three model calls; the worst case is ten and then it stops. There is no unbounded agent loop anywhere in it. Nothing a model wrote is trusted on its word. All of it lands in tests/generated/ , a gitignored scratch directory that your normal test run, your typecheck, and your linter never look at. A human reads the diff, and, more to the point, watches it run: npm run review opens the scratch suite in Playwright's UI mode, where you step through each test and see what every locator actually matched and whether the primary assertion lines up with what is on screen. If it holds, npm run freeze moves the file into the baseline. That move is the single step in the whole system with no automation behind it, and it stays that way. Generation is a dev-time activity. Once a spec is frozen, CI never calls a model again. It runs plain deterministic Playwright and a mutation gate. No agent in the build, no token bill per push, no model deciding whether your release is green. Keeping the model in that hot path, the way self-healing CI and in-pipeline test generation do, buys you a class of red build you cannot reproduce, because the thing that failed was a sampling temperature, not your app. Not on its word. The frozen baseline goes through the same gate as the first two posts, playwright-mutation-gate https://www.npmjs.com/package/playwright-mutation-gate On the bundled demo shop the pipeline generated assertions for every feature, and the gate killed all fifteen. No hollow greens in the generated baseline. 15 killed, 0 survived, 0 cannot-mutate, 0 errors Gate passed: every marked assertion can fail. The repo also keeps one deliberately weak spec outside the trusted baseline: a receipt greeting drawn from a name the browser cached at checkout rather than from the order the server confirmed. Inversion clears it, because the greeting really is on screen. The gate's --behavior pass corrupts the value in the server response and catches it staying green anyway. That is the exact wrong-source-of-truth trap the second post https://dmitriiev.dev/posts/passing-tests-never-talk-to-the-server/ is about, kept runnable next to the pipeline that would otherwise let it through. The demo shop is small and server-rendered, so its whole source fits in one prompt. A real app needs a curated slice of the frontend, the templates and components for the flows under test, pointed at through src/pipeline.ts , plus a hand-written reference spec for the house style and stable data-testid attributes on the elements your flows touch. The agents treat that source as ground truth, which is what keeps them from inventing selectors. There is no black-box mode that works from a rendered DOM instead, and there will not be one until it can keep that guarantee. A few things this deliberately leaves on the table. A pipeline whose whole job is to earn trust cannot pretend they are not there. Curating that frontend slice is real work. On a large app it is the main cost of adoption, more than anything in the pipeline itself, and feed it the wrong components and the writer is back to guessing. Generated specs match the house style of the one reference spec you hand the writer, but they do not wire themselves into your existing Page Objects or shared helpers. On a mature suite with its own abstractions, that reference spec has to carry them, or you refactor after freezing. The pipeline writes tests that look like your best hand-written one; it does not know your framework. Frozen specs are ordinary Playwright tests once they land, and they rot like any others. A renamed data-testid or a redesigned flow breaks them, and someone maintains them by hand or regenerates. This solves the blank-page-and-trust problem, not the lifecycle after it. Test data, auth, and seeded state are on you. The feature files describe user actions; getting the app into the state to run them is fixtures you write, the same as any E2E suite. And the deepest check is stack-shaped. The mutation gate's --behavior pass, the one that proves an assertion is bound to the server rather than to a cached copy, only sees values that cross the wire as text. It bites on server-rendered pages and verbatim APIs and goes quiet on strings a client composes in the browser the second post https://dmitriiev.dev/posts/passing-tests-never-talk-to-the-server/ has the detail . The pipeline still generates and runs tests against a React app; you just get less of that one guarantee exactly where the values are assembled client-side. Clone it, run the pipeline against the demo shop, and watch a spec earn its way into the baseline: ai-qa-pipeline https://github.com/VladyslavDmitriiev/ai-qa-pipeline . I build pipelines like this for teams. If your suite is green and you are not sure what that buys you, that is the conversation I have at automation.dmitriiev.dev https://automation.dmitriiev.dev .