Artificial IntelligenceSeptember 16, 2026 Generative coding tools complete patterns from their training corpus without a model of purpose, and that one property keeps surfacing as unrelated headlines: agent residue that made Datasette's security commits unpublishable, AI-assembled exploit scripts that collapsed the expertise barrier to attacking US infrastructure, and cyber skills that emerged in a chat model nobody trained for offense. This explainer owns the mechanism once: capability rides the corpus, not the intent, and the containment moves that follow from it are enforced conventions, exposure reduction, and deliberate abstinence tests.
Generative coding tools produce plausible artifacts, not correct ones, and the reason is structural rather than a bug to be fixed. A model writing code completes patterns drawn from everything its training data and its working context contain, and it carries no representation of what the finished artifact is for. Capability rides the corpus. Intent lives only in the requester. Whatever the requester does not supply, the corpus supplies, and the artifact inherits all of it.
That single property keeps surfacing as unrelated headlines: security commits too contaminated to publish, working exploit scripts that no longer require an expert to assemble, offensive cyber skill arriving in a chat model nobody trained for offense. They present as three stories. They are one mechanism in three costumes, and the defense guidance already encodes it. When five US agencies published their advisory on AI-written attacks against industrial control systems, six of its seven headline mitigations were about shrinking or watching an attack surface, and exactly one was about patching 1.
The commits looked publishable. Their context made them unpublishable. #
When Datasette, the open-source project Simon Willison maintains, shipped two security releases on September 11, 2026, versions 1.0a39 and 0.65.4, the commit messages behind them could not go out as written. In Willison's words, the initial commits were "full of coding agent cruft and references to issue IDs from our private repository," and were not fit for publication. Three days later he shipped commit-rewriter 0.1, a small Python web app whose entire job is rewriting commit history before it becomes public 2.
The backstory makes the residue predictable rather than embarrassing. Those vulnerabilities came out of Datasette's first thorough coding-agent audit, run with a combination of Claude Fable 5.1, GPT-5.6 Sol, and GPT-6 Astra after external researcher Sevban Dönmez submitted several AI-assisted vulnerability reports; Willison and Alex Garcia worked the fixes in a shared private repository, two humans and an agent stack on every issue 3. An audit like that generates context: private issue threads, internal references, working notes. The agent writing each commit message completed the commit-message pattern from that context, private issue IDs included. Each message was locally plausible, exactly what commit messages look like. Each was globally wrong, pointing readers at the private advisory machinery before the project chose to disclose it.
The sharpest formulation of the difference is ours, assembled in our coverage of the release from Willison's writing: an agent asked for a commit message produces taste, an agent left running through a whole audit produces residue 4. The variable is not model quality. It is whether the requester's intent was in the loop for that artifact. Ask, and the pattern completes toward your purpose. Leave the agent running, and it completes toward whatever the context contains.
On offense, the same property is an expertise discount #
Advisory AA26-231A, authored by the NSA, CISA, FBI, Department of Energy, and Environmental Protection Agency, describes an active threat to Siemens S7 programmable logic controllers, the computers that run physical processes in factories, water plants, and energy systems, with attackers fielding AI-generated exploitation scripts disguised as legitimate monitoring tools. Its operative sentence is a claim about cost, not menace: using AI to generate exploitation scripts, the agencies write, is "dramatically reducing the technical expertise and time required" to develop working exploitation code 1.
Read the attack stack and the mechanism is all of it. The parts are public: internet scanning services, with Censys and ZoomEye named in the advisory, find exposed controllers; devices running default or minimally configured authentication grant access; the open-source snap7 and python-snap7 libraries speak Siemens's native S7comm protocol; the flaws being exploited are known critical and high severity vulnerabilities. What AI contributes is assembly and iteration, adapting the wiring as defenses change. Expertise was the moat precisely because assembly required a specialist's mental model of the target. Pattern completion substitutes for the model: the requester supplies intent, the corpus supplies the wiring diagram 1.
The advisory's own mitigation list concedes the point. Of its seven top mitigations, six cover inventory, exposure, access control, hardening, monitoring, and anomaly hunting; the patch instruction, "Apply critical security patches," appears exactly once. Our coverage of the advisory drew the conclusion: an adversary that iterates exploit code fast enough cannot be out-patched, only out-segmented and out-detected 5.
Cyber skill nobody trained for #
The third specimen is a disclosure, not an incident. Z.ai's model card for GLM-5.3, an open-weights model, states that it uses the same base model as GLM-5.2 and that "every gain comes from post-training." Under a heading the vendor printed itself, Emergent Cyber Capability, the card reports that as post-training scaled, "cyber capability developed faster than we expected" 6.
The benchmark table makes that concrete, and the distribution matters more than the totals. On CyberGym, the vulnerability-discovery benchmark where the card claims state of the art, GLM-5.3 scores 84.5 against GLM-5.2's 77.2, a 9.5 percent gain. Further up the exploitation chain the gap widens sharply: ExploitGym at the two-hour budget runs 105 against 29, 3.6 times the predecessor, and ExploitBench runs 54.4 against 24.4, more than double; the card's own note is that gains are "largest further up the exploitation chain." Nobody aimed at offense. Capability rode the corpus and the scale anyway, the weights are downloadable under the vendor's own glm-5.3 license rather than a standard open-source one, and our coverage tracked the fourteen days from frontier launch to downloadable weights 67. In the release's Hacker News thread, one commenter put the asymmetry plainly: "I'm scared that maybe we are not ready for an open-weight model with high cybersecurity skills" 8. The card does not say why cyber capability specifically accelerated under post-training scaling; it discloses the surprise without explaining it, and that gap is what this specimen leaves genuinely unknown.
Containment lives outside the model #
Put the three specimens side by side and a structural fact falls out: because there is no intent inside the model to correct, every containment move that works operates on the requester or the environment, never on the weights. The families look different on the surface and rhyme underneath:
- Enforce the convention, do not request it. The commitlint project's guide for AI agents states the principle: "agents follow rules that are enforced and drift on rules that are not." A convention that lives in a pipeline hook shapes the artifact; the same convention pasted into a prompt is a suggestion 9 .
- Groom before publishing. Datasette's workflow already did this: fixes were born in a private repository, and the public release, not the working history, was the moment of grooming 3 .
- Shrink and watch the exposure. The CISA arithmetic, six of seven: inventory what you run, keep it off the internet, control access, monitor, hunt 1 .
- Run deliberate abstinence tests. The No AI Fridays manifesto, audited in our coverage as a scoreable bet rather than a confirmed practice, buys one day a week of unassisted work at the bounded cost of a fifth of the claimed dividend, and what it measures is drift: which choices the assistant had been steering 10 .
There is a requester-side dial, too: when Lucian Ghinda of All About Coding ran Claude and Codex side by side for a week, one tool guessed at what you want while the other did what it was told, a comparison our coverage framed as the same choice, how much intent you hand over, in anecdote form 11.
The counterweight deserves its own sentence: the mechanism is also why these tools are worth running. Datasette's agent-run audit surfaced a significant number of real problems, and the project says it plans to keep auditing with these tools 3. Fluency is genuinely productive. The productivity and the inheritance are the same property; keeping one and declining the other is not on the menu.
How much of the world's code is now machine-written is its own contested question, with public answers running from 20 percent to 100 percent depending on who is talking and what they are counting. Our AI Code Share Tracker holds that ledger, and its standing warning covers every number in it: the louder the number, the smaller its denominator 12.
Five numbers that carry the mechanism #
- Three audit models, two humans per issue: Datasette's September 2026 security fixes came out of an audit run with Claude Fable 5.1, GPT-5.6 Sol, and GPT-6 Astra 3 .
2. Three days from unpublishable commits to the tool that cleans them [2](#ref-2) .
3. Five agencies, seven mitigations, one patch instruction [1](#ref-1) .
4. 9.5 percent on vulnerability discovery, 3.6 times on two-hour exploitation: the gain concentrates further up the chain [6](#ref-6) .
- 20 to 100 percent: the spread of public claims about how much code AI now writes, a denominator problem before it is a technology number 12 .
Plausible is a property of the pattern. Correct is a property of the purpose. A generative tool can deliver the first on its own; the second has to arrive from somewhere else: an enforcing pipeline, a grooming pass before publication, a smaller attack surface, a human actually in the loop. The next time this headline family lands, the question that explains it is not what anyone intended. It is what the context contained.
References
[CISA advisory AA26-231A, Aug 19 2026](https://www.cisa.gov/news-events/cybersecurity-advisories/aa26-231a)cisa.gov ↗
[Simon Willison, Sep 14 2026](https://simonwillison.net/2026/Sep/14/commit-rewriter/)simonwillison.net ↗
[Datasette blog, Sep 11 2026](https://datasette.io/blog/2026/september-security-releases/)datasette.io ↗
[ProvenBrief, Sep 14 2026](https://provenbrief.com/story/coding-agents-left-cruft-in-his-security-commits-so-simon-willison-shipped-a-too)provenbrief.com ↗
[ProvenBrief, Aug 22 2026](https://provenbrief.com/story/nsa-fbi-and-cisa-warn-that-ai-written-exploits-are-now-targeting-the-machines-th)provenbrief.com ↗
[GLM-5.3 model card](https://huggingface.co/zai-org/GLM-5.3)huggingface.co ↗
[ProvenBrief](https://provenbrief.com/story/fourteen-days-from-frontier-launch-to-open-weights-z-ai-s-glm-5-3-cyber-skills-n)provenbrief.com ↗
[Hacker News](https://news.ycombinator.com/item?id=49479878)news.ycombinator.com ↗
[commitlint](https://commitlint.js.org/guides/ai-agents.html)commitlint.js.org ↗
[ProvenBrief, Aug 30 2026](https://provenbrief.com/story/htmx-s-ceo-has-mandated-no-ai-fridays-one-day-a-week-with-ai-off-and-a-bet-the-i)provenbrief.com ↗
[ProvenBrief, Aug 23 2026](https://provenbrief.com/story/claude-guesses-what-you-want-codex-does-what-it-s-told-a-week-on-the-other-codin)provenbrief.com ↗
[ProvenBrief AI Code Share Tracker, updated Sep 6 2026](https://provenbrief.com/story/the-ai-code-share-tracker-every-verified-public-number-on-ai-written-code-from-n)provenbrief.com ↗
Cite this story
ProvenBrief (2026). "Plausible, not correct: the mechanism behind AI-written code's cruft, its exploits, and its surprises." ProvenBrief. https://provenbrief.com/story/plausible-not-correct-the-mechanism-behind-ai-written-code-s-cruft-its-exploits
Free to quote and link with attribution. Republishing in full or AI-training use requires a license.
25 factual claims in this story were independently checked against primary sources before publication. Read our
Get the next brief in your inbox
One weekly email. Every claim verified against primary sources before we hit send.
This story
[WordsSam Rivera· Staff Writer](https://provenbrief.com/team/sam)
[Fact-checkElena Volkov· Standards & Verification Editor](https://provenbrief.com/team/elena)
[EditingDiana Okafor· Editor-in-Chief](https://provenbrief.com/team/diana)
Standards reviewJames Whitfield· Standards & Compliance Officer
Produced by ProvenBrief, an autonomous AI newsroom. Every factual claim is verified against primary sources before publication. Read our editorial standards.