Why Agentic Coding will Always Introduce Errors

Agentic coding will always introduce errors because large language models generate statistically plausible patterns rather than system-aware modifications, making semantic mistakes unavoidable even with strong agentic reasoning, according to an analysis of transformer design. The article argues that only human-authored tests and architectural invariants can prevent these errors from entering production, as LLMs are not designed to infer or maintain the structure of pre-existing codebases.

When using an LLM to modify pre-existing code, it is not possible to get the error rate to zero. This article shows why agentic coding will always introduce errors because LLMs generate statistically plausible patterns rather than system‑aware modifications, making semantic mistakes unavoidable even with strong agentic reasoning. Only human‑authored tests and architectural invariants can prevent these errors from entering production. Why code modification is hard Code modification is hard because the LLM is not solely operating inside its own learned patterns as it is when it produces greenfield code based just on a prompt. To get code modification correct, the LLM must operate inside the patterns of the codebase. Code modification requires a transformer to work with the codebase which is a structure it did not learn during training, and transformers are not designed to infer, maintain, or reason about such structure. They are built to continue patterns, not model systems. Transformer design Transformers are designed to model sequences using self‑attention, a mechanism that statistically captures relationships between positions regardless of what the sequence represents. In natural language, this works extremely well because English contains stable statistical patterns. These patterns become apparent when processing vast amounts of text: "dogs run" is common, "dog runs" is common, but "dogs runs" is not. A transformer does not know anything about plurality or grammatical agreement; its text processing simply shows that certain token pairings occur frequently and others almost never do. It is the statistically common case that makes modelling the English language with self-attention reliable. Programming languages exhibit similarly common usage patterns. For example: if condition { do something } else { do something else } appears extremely often in training data, whereas: if condition { do something } else do something else } almost never appears. Processing vast amounts of correct code gives the transformer the ability to generate correct code because correct usage is overwhelmingly more common than incorrect usage. When the model predicts the next token, it is statistically biased toward the patterns that appear most frequently. And in programming languages, those frequent patterns correspond to syntactically valid code. The LLM does not understand the syntax of a programming language nor does it model programming language grammar or sementics. The LLM does not understand the semantics of an if-else, i.e., that only one branch will be executed, and the LLM cannot determine what condition will be only by looking at the text. An LLM simply learns that valid token sequences like if condition appear vastly more often than invalid ones like if condition . And because correct usage dominates the training distribution, next‑token prediction is statistically biased toward the more common patterns — and those common patterns happen to correspond to syntactically valid programming code. Correctness is a by-product of statistics, not understanding. But code modification requires more, beyond bias towards common patterns. An example Imagine you have read through a codebase and you notice that all writes to user accounts are not made directly but via a wrapper. You can see that the wrapper performs audit logging. This approach is not written down anywhere. It is enforced by the architecture and historical use of the codebase, as well as by team practice. To safely modify this code, any change must continue to respect that all writes go via the wrapper for audit logging compliance. Based on its training data, statistical patterns and next token prediction, the LLM generates: account.balance += amount because this is a high-probability token continuation. But this code breaks the system as the update of 'account' does not go via the wrapper. The generated text is correct but what it means is incorrect with respect to the system that text must operate within. Fundamental difference Pattern-continuation solves this: From my training data, I have seen many examples of updating a balance; here is a plausible one Safe code update requires: In this system, balance updates must go through a wrapper because of a compliance rule that is not explictly represented anywhere in text. When it comes to modifying pre-existing code, the model can only suggest a likely next token. The model is unaware it should only generate tokens that respect the use of the wrapper. Safe code modification requires respecting constraints that are not present in LLM training patterns. There may be constraint respected examples in its training data but the transformer has not been designed to statistically model them. To the transformner, everything is just text, there are no domain semantics. The transformer models patterns in text with no regard to what that text means. Error can never be zero Because transformers generate text by continuing statistically likely patterns, and because system‑level constraints are not represented in those statistics, no amount of prompting or additional context can eliminate constraint‑violating continuations. Error cannot go to zero. Prompting and additional context can reduce the error but will not eliminate it. Agentic systems Agentic systems add software engineering-specifc tooling to an LLM system. Such systems operate a workflow of: LLM proposes → verifier checks → tool executes. Before any LLM proposed action is executed, a number of verifiers are used. These are rule based and perform static programming language checks, schema validation, safety filtering and the enforcement of invariants. The verifiers can only check explicit programming language semantics: types, schemas, pre- and post-conditions. Verifiers cannot check meaning that is not formally expressed. Any semantics that are implicit, domain-specific, or are derived from "that is how we do things" cannot be checked. In any hybrid agentic system, the LLM can output a semantically wrong action, and no amount of tooling, verification, or scaffolding can eliminate that class of error. Hybrid systems help reduce the mechanical error rate but semantic errors can never be eliminated. But this is OK. To err is LLM. Agentic failure modes The work of Albayaydh–Zhao–Flechais 2026 in their "leaderboard paper" is a synthesis of 27 other works from between 2023 to 2026, covering 19 benchmarks. Their synthesis has produced these categories of agentic failure: - Tool‑invocation and parameter‑level errors - Planning and constraint‑satisfaction failures - Long‑horizon degradation from context accumulation - Multi‑agent coordination failures - Safety failures unsafe agent behaviour - Security failures external adversarial conditions - Measurement validity problems Points 6 and 7 are not caused by the LLM. They are to do with conditions external to the agent. Points 1 to 5 are caused by the probabilistic nature of the LLM. Probabilistic token generation is an approximation. Given this, LLM-centred agentic software development will never be fault free. Agentic software development lifecycle A software engineer using an agentic system has to: - shape prompts - review agent output - correct LLM mistakes due to hallucinations or the semantic gap - run tests and based on this the engineer must reprompt the LLM or retry some tooling. Eventually, an LLM- and engineer-developed artefact will be generated that the engineer will be happy to integrate into a production system, while also ensuring architectural consistency and guarding again silent errors: both of which may require engineer-performed direct changes to the artefact. In practice, agentic workflows reduce construction time but increase review and validation time. Agentic systems introduce new overhead Even if the agent produces good artefacts, teams must maintain: - agent behaviours - prompt libraries - safety rails - integration harnesses - test suites designed for agents This is additional overhead that was not required before agentic systems. An agentic software development lifecycle requires an engineer at the centre scheduling tasks, orchestrating and conducting workflow, reviewing outcomes and validating them for subsequent deployment. Human involvement is mandatory because LLM-centred tools make mistakes. Is Agentic development about speed? No. Using an agentic system to push for speed, creates pressure to ship faster by cutting corners: reducing review time, accepting lower-quality code, and reducing testing. This will have the effect of pushing features out more quickly but also prematurely. More is not better. Software development is not solely about code generation. It is mostly about understanding the requirement to ensure that the right code is written, regardless of whether the code is LLM-generated or human authored. Pushing for faster shipping is an approach that will affect the whole business: - product more time spent triaging issues instead of contributing to building - design rework due to unstable requirements - support increased customer issues - sales prematurely released features break or regress - marketing launches are undermined by defects - finance the higher cost of incidents and rework due to premature code release - brand trust erosion from unstable releases - board confidence undermined Agentic development only makes economic sense if the final artefact is at least as good as the human‑only artefact. If agentic development leads to a reduction in the quality of the final artefact, businesses will pay the cost later in increased rework, more production issues, and elevated maintenance costs. Rework will correct the immediate defect. Maintenance cost reflects the long-term cost of poor-quality artefacts that adversely affect system structure and increase test fragility, both of which represent regression risk lt leaves the business with less room for manoeuver when needing to change code. Quality reduction has a knock-on effect resulting in a long-tail of consequences. Quality is king When considering agentic development, the only thing that matters is: Does the human plus AI workflow produce a higher‑quality artefact than a human alone? At a minimum, does the agentic approach produce equal quality at lower cost? There is no general answer to either of these. They can only be answered accurately with respect to the context of a company: their specific agentic software development lifecycle, and the company's approach to testing, confirmation, verification, and release. Errors will always be with us Before agentic development, a quality outcome was already the goal of every software delivery pipeline. Agentic development does not change this goal, it only changes how artefacts are produced. Both humans and LLM‑centred agentic tooling produce errors. LLMs introduce new classes of mistake: hallucinations, semantic gaps, mis‑inferred invariants, and silent failures. Humans introduce their own: misunderstandings, typos, incorrect assumptions, and incomplete reasoning. Therefore, an agentic software development lifecycle must include testing. Human‑generated artefacts require tests; so do artefacts generated agentically. And critically: those tests must be solely human‑authored. If tests are generated by the same agentic system that produced the code, the tests may simply reproduce the same misunderstandings or hallucinations. Errors can hide inside the volume of generated code and generated tests, reinforcing each other rather than revealing faults. Only human‑authored tests provide an independent, trustworthy oracle of correctness. Testing all calls are via a wrapper Agentic development requires a new kind of test: tests that assert architectural invariants, not just functional behaviour. If such a test is not written, eventually, the agent will by-pass the wrapper. This test must assert a fact about the structure of the system, not test its logic. It is a test that confirms something about how the system is built. This kind of code would need to build an abstract syntax tree of the codebase, and then test whether every call in that tree the whole codebase , all go via the wrapper. Conclusion Agentic output will never be perfect. But that is OK. Results only need to be correct-enough that a human can efficiently get the code to 100%, where overall cost is cheaper than the human doing 100% of the work. Outline Python to build an AST The AI-generated code to build an abstract syntax tree from a .ZIP URL is short, only 51 lines. You are free to use this code. No warranty is expressly given or implied. You are using this code "as-is" and you should check its applicability for use within your systems and Python environment. import ast import pathlib import tempfile import urllib.request import zipfile import tarfile def download and extract url: str - pathlib.Path: """Download a codebase archive from a URL and extract it.""" tmpdir = pathlib.Path tempfile.mkdtemp archive path = tmpdir / "codebase" Download the archive urllib.request.urlretrieve url, archive path Extract depending on format if zipfile.is zipfile archive path : with zipfile.ZipFile archive path, "r" as z: z.extractall tmpdir elif tarfile.is tarfile archive path : with tarfile.open archive path, "r: " as t: t.extractall tmpdir else: raise ValueError "Unsupported archive format" return tmpdir def build ast for codebase root: pathlib.Path : """Parse every .py file under root into an AST.""" ast map = {} for file in root.rglob " .py" : try: source = file.read text tree = ast.parse source, filename=str file ast map file = tree except Exception as e: print f"Failed to parse {file}: {e}" return ast map Example usage: if name == " main ": url = "https://example.com/my project.zip" your codebase URL root = download and extract url ast map = build ast for codebase root print f"Parsed {len ast map } Python files into ASTs." This code gives you: - a directory containing the full codebase - a dictionary mapping the .ZIP file to an AST - a typed AST you can traverse to assert architectural invariants The code below provides a test that traverses the AST: python def find calls tree, module, func : calls = for node in ast.walk tree : The node in the if is a function call and the code is modelled as a Python attribute so this code matches method invocations of the form: object.method ... if isinstance node, ast.Call and isinstance node.func, ast.Attribute : if node.func.attr == func and getattr node.func.value, "id", None == module: calls.append node return calls def test no direct user updates ast map : violations = for file, tree in ast map.items : if account.update found, wrapper not used bad = find calls tree, "account", "update" if bad: violations.append file, bad assert violations == , f"Direct calls found: {violations}" Read next: Vibe Coding Is Not Engineering LLMs generate code, but they cannot see the engineering decisions that keep systems safe. Related Articles Why Junior Engineers Matter More as AI Expands /articles/build/notes/why-junior-engineers-matter-more.html The Myth of Complete Specifications /articles/build/notes/the-myth-of-complete-specifications.html The Limits of Stateless LLMs /articles/build/notes/the-limits-of-stateless-llms.html What Software Engineers Need to Know About LLMs /articles/build/notes/software-engineers-need-to-know.html Evaluating AI Systems: Metrics that Matter /articles/build/notes/evaluate-ai.html If this was useful , you can get more pieces like it in the Phroneses newsletter.