{"slug": "red-green-refactor-was-always-a-provenance-rule", "title": "Red-Green-Refactor Was Always a Provenance Rule", "summary": "Two 2026 studies (arXiv 2607.05139 by Konstantinou, Tambon and Papadakis, and arXiv 2609.09315) measured how test-authoring order affects fault detection, finding that tests generated from a task description alone catch roughly 25% of injected faults, while tests written after the model has seen the implementation catch only about 14%, a gap significant at p < 0.05. The second paper reproduced the same relative pattern with lower absolute rates of roughly 3% to 8%. The analysis argues that red-green-refactor's value lies in test provenance: an oracle derived from the specification outperforms one contaminated by the implementation under test, a property human developers enforced implicitly by writing tests before code existed.", "body_md": "*Originally published at [tddbuddy.com](https://www.tddbuddy.com/blog/red-green-refactor-is-provenance/).*\n\n*Related reading: [The Contract Test Is the Only Witness the Agent Cannot Author](https://www.tddbuddy.com/blog/the-contract-test-is-the-only-witness/) named the same-author problem at the integration seam. [The Instruction File Is Not the Discipline](https://www.tddbuddy.com/blog/instruction-files-are-not-the-discipline/) named why prompt rules do not reach this failure mode. [Agents Should Do TDD](https://www.tddbuddy.com/blog/agents-should-do-tdd/) argued the loop converges regardless of who runs it; this post pins the specific property of the loop that an agentic workflow has to encode structurally to inherit the convergence.*\n\nThe agent wrote the implementation, then wrote the test, then reported a green bar. The injected fault shipped.\n\nFor twenty years the three-step loop was taught as workflow. Write the failing test, make it pass, refactor. The sequence read as style, a preference developers held when they were disciplined and skipped when the ticket was due. Nobody wrote down why the order mattered, because the human loop enforced it mechanically. A developer at a keyboard reading the ticket wrote the test before the code existed. There was no other way to run the loop.\n\nTwo 2026 studies pin a different reading. The temporal order was never a style choice. It was a provenance claim about the test: that the oracle encoded in the assertion is derived from the specification, not from the artifact under test. When the implementation is in the author's context before the test is written, the test inherits the implementation's blind spots. The green bar it produces is a tautology dressed as verification.\n\nRed-green-refactor was never about discipline-for-its-own-sake. It was about where the oracle came from.\n\nThe literature taught the loop as a procedure a developer runs. Beck's examples alternate between red, green and refactor the way a tennis coach alternates between forehand and backhand drills, as if the pedagogy were the point. Teams that adopted TDD talked about it as hygiene: write the test first, resist the temptation to skip it, discipline yourself through the deadline.\n\nThe reason the output was better was quietly structural. A test written before the code it validates is an oracle derived from one source: the specification the author read when they wrote the test. A test written after the code it validates is an oracle derived from two sources: the specification, and the artifact that is now sitting in the author's working memory as a plausible answer. The second source contaminates the first. Any fault the artifact already carries propagates into the oracle, which then certifies the fault as correct.\n\nHumans never had to name this property because the human loop enforced it by physics. There was no implementation to contaminate the oracle with, because there was no implementation yet. The provenance rule rode on the back of typing speed.\n\nKonstantinou, Tambon and Papadakis (\"On the risk of coding before testing\", arxiv 2607.05139) measure fault-detection rate as a function of test authoring order. Tests generated from the task description alone detect roughly 25% of injected faults. Tests generated after the model has seen the implementation detect roughly 14%. The gap holds at p < 0.05 across the benchmarks they ran. A separate paper (arxiv 2609.09315) measures oracle-generation fault detection in a different setup and lands lower absolute numbers (roughly 3% to 8% across benchmarks) with the same relative shape. Oracle derived from the specification outperforms oracle derived from the faulty implementation. The mechanism reproduces across experimental conditions.\n\nThe results are empirical confirmation of a property the human discipline encoded without naming. The oracle has a source. The source has a quality. The quality of the oracle is bounded above by the quality of the source. When the source is a faulty implementation, the oracle inherits the fault and then certifies it.\n\nAn informed reader will reach for the Vathana result (\"LLM vs. Human Unit Tests\", arxiv 2606.08588), which reports retrieval-augmented LLM tests detecting faults at 69% versus 17.2% for a general-purpose human baseline. The result does not refute the provenance claim; it validates it. The retrieval context in that study is the specification channel, surfaced to the model as augmented input. What the Vathana paper measures is that an oracle derived from the specification plus relevant retrieval outperforms an oracle derived from neither. That is the provenance rule, re-expressed. The post concedes the finding and sharpens the claim: the failure mode is specifically code-first oracle derivation, which the Konstantinou numbers measure directly and which the Vathana setup does not test.\n\nThe field reached for chain-of-thought and chain-of-verification prompting as a procedural fix. If the model reasons harder before writing the test, surely it catches the implementation's bugs. The numbers say otherwise. CoT and CoV prompting do not close the 25-to-14 gap, because the fault is already present in the input the oracle is derived from. More reasoning tokens on a corrupted context compound the certainty of the wrong answer, not the probability of catching the fault.\n\nThis is the structural result the prompt-level interventions keep running into. A prompt saying \"do TDD\" does not change which signal shaped the oracle, because the agent is still looking at the implementation when it writes the test. The reasoning chain the model unspools reads the implementation, forms a plausible explanation of what the implementation does, and derives an oracle that matches the explanation. The oracle agrees with the implementation because the oracle was authored by a process that spent its reasoning budget rationalizing the implementation.\n\nThe mitigation has to live outside the prompt. No amount of instruction-tuning on the writer closes the gap, because the writer's problem is not instruction; it is context.\n\nGenerate the test from the specification. Keep the implementation out of the context window until after the test is authored. Enforce the order at the workflow layer, not the prompt layer. The suite's authoring order is a property of the pipeline, not a convention of the writer.\n\nA concrete shape: a two-stage generation pipeline. The first stage sees the ticket, the builder catalogue, and the existing scenario index. It produces the test. The implementation path is masked from this stage's context. The second stage sees the test and the production code tree. It produces the implementation against the fixed test. A third gate fails closed if the test file was modified during the implementation stage. The order is a guarantee the pipeline enforces, not a convention the agent is asked to follow.\n\n```\n[Fact]\npublic void A_loyalty_member_with_an_order_over_50_dollars_receives_a_ten_percent_discount()\n{\n    var order = anOrderOver(50.dollars())\n        .forCustomer(aLoyaltyMember());\n\n    var receipt = checkout.process(order);\n\n    receipt.discount.Should().Be(6.dollars());\n}\n```\n\nSuite A, derived from the specification only, produces that test. The assertion pins the behavior named in the ticket. Suite B, derived from the implementation after the agent writes it, produces the same scenario with an assertion that mirrors whatever the implementation returned, including the off-by-one in the threshold case the implementation got wrong. Suite A fails on the off-by-one and the agent fixes the implementation. Suite B certifies the off-by-one and the agent ships it. Both agents followed instructions. Only one of them was handed a context that could catch the fault.\n\nThe pipeline shape matters more than the model behind it. A stronger model given the full context in one pass still writes the receipt. A weaker model given the two-stage context writes the test.\n\nA test records what the code does. A test derived from the implementation records what the implementation already did. The distinction looks pedantic until the implementation carries a fault, at which point the first category catches it and the second category certifies it.\n\nThe green bar a receipt produces is a tautology dressed as evidence. The agent that writes it is faithfully reporting the artifact under test; the suite that accepts it is accepting the artifact's self-report as verification. This is the shape the Konstantinou numbers measure. A test author with the implementation in context converges on an oracle that agrees with the implementation, which is the opposite of what verification is supposed to do.\n\nThe oracle has to come from somewhere other than the thing it is checking. That is the whole point of the oracle. Any workflow that collapses the two sources into one hands the author a mirror and asks it to find its own blind spot.\n\nThe contract-test post named the same-author problem at the integration seam. Two mirrors facing each other produce a green bar on an interface neither has verified against the counterparty. The mitigation there is a second author with independent context: the consumer publishes a contract on infrastructure the provider's agent cannot rewrite. Separation of authorship is enforced by the repository layout.\n\nThe provenance rule is the same principle, applied at the unit tier, inside one author. There is no second agent at the unit level; there is only the one agent running the loop. The lever that stays available at that tier is time. The oracle is written while the implementation does not yet exist. The author cannot see the artifact under test, because the artifact under test has not been written. The mirror cannot form, because there is nothing on the other side of it yet.\n\nBoth posts share a mechanism. The writer cannot edit the checker, and the writer cannot have already read the thing being checked. At the seam, that property is enforced spatially: two trees, two authors. At the unit, that property is enforced temporally: one tree, two moments. Governance in both cases is architectural. The prompt that says \"do TDD\" is not the fence. The pipeline that generates the test from a context without the implementation in it is the fence.\n\nA rules file documents the intent. A pipeline enforces the oracle's provenance. The two artifacts are not interchangeable, and the Konstantinou numbers are the receipt for which one does the work.\n\nThe human loop enforced provenance by physics. The agentic loop enforces nothing by default: both the test and the implementation are generated from the same context in the same pass, and the order the discipline depended on evaporates unless the pipeline reinstates it. The agent that wrote the implementation, then wrote the test, then reported a green bar did everything it was asked to do. The green bar was a tautology, because the oracle and the artifact under test came from the same source. The fault shipped not because the agent failed, but because the workflow did not encode the property the discipline was always carrying.\n\nRed-green-refactor was never a workflow preference. It was a provenance claim the human loop made for free and the agent era has to pay for in pipeline configuration. The order was always the property. The property was always the point.", "url": "https://wpnews.pro/news/red-green-refactor-was-always-a-provenance-rule", "canonical_source": "https://dev.to/tmfrisinger/red-green-refactor-was-always-a-provenance-rule-2ll9", "published_at": "2026-11-05 21:16:28+00:00", "updated_at": "2026-11-05 21:23:25.879679+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-safety", "developer-tools"], "entities": ["Konstantinou", "Tambon", "Papadakis", "arXiv", "TDD Buddy"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/red-green-refactor-was-always-a-provenance-rule", "markdown": "https://wpnews.pro/news/red-green-refactor-was-always-a-provenance-rule.md", "text": "https://wpnews.pro/news/red-green-refactor-was-always-a-provenance-rule.txt", "jsonld": "https://wpnews.pro/news/red-green-refactor-was-always-a-provenance-rule.jsonld"}}