# The Memory Wall

> Source: <https://dev.to/diyaburman/the-memory-wall-25g7>
> Published: 2026-08-03 13:30:00+00:00

I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.

Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. [danshapiro.com](https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory)

Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. [natebjones.com](https://www.natebjones.com/) — [Watch the video](https://youtu.be/bDcgHzCBgmQ)

This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.

Layer 3 begins here.

Twelve issues of building: a working API, a Gherkin test suite, Pact contracts, a CI/CD pipeline, and a skill layer with six active skills across three tiers. More infrastructure than most projects of this scope ever build.

It is not enough.

This issue is about why. Not in the abstract — in the specific, mechanistic sense. What is the exact chain of events that turns missing context into a production failure? And what class of failure does the specification layer miss, and the skill layer miss, that Layer 3 exists to prevent?

Three data points. Three different methodologies. The same mechanism underneath each one.

**The 2.5% completion rate**

The Remote Labor Index study measured AI agent performance on real-world software freelance tasks drawn from Upwork — tasks with actual pay, actual clients, and actual acceptance criteria. The headline: agents completed approximately 2.5% of tasks successfully.

What "failure" means here is not a crash or an exception. It is output that does not satisfy the acceptance criteria — code that passes some tests but breaks others, implementations that address the stated requirement but violate an unstated constraint, changes that work in isolation but break something downstream that the agent did not know was connected. The agents did not fail to produce output. They failed to produce output a human reviewer would accept.

What the successful 2.5% tasks have in common: they are self-contained. The context needed to succeed is present at the point of invocation. The failing 97.5% require context that exists outside the task description — why a previous decision was made, what other systems depend on the code being changed, which properties of the current implementation are invariants and which are incidental.

**The 95% enterprise pilot failure rate**

McKinsey, Gartner, and MIT Sloan research converge: the majority of enterprise AI pilots do not reach production deployment. Not that they produced nothing useful. Many produced impressive demos and working prototypes. "Fail to reach production" means the system was never deployed to serve real customers.

The transition from "works in the pilot" to "runs in production" is where the failures cluster. The primary reasons, in order: data and context problems, security and compliance requirements that the pilot never encountered, integration brittleness against real external systems, and undocumented invariants that the pilot broke without knowing they existed.

This is a context problem, not a capability problem. The capability to write working code and integrate with external APIs is present in the pilots — the demos show it. What the pilots lack is the context to make decisions that align with constraints outside the pilot environment: legal constraints, security invariants, integration contracts, operational history.

**The productivity paradox revisited**

Issue #2 documented the finding: experienced developers using AI tools took 19% longer while believing they were 24% faster. Revisiting this through the lens of Layer 3: this is not primarily a skill problem or a spec problem. It is a stewardship problem.

The 19% slowdown is concentrated in integration work, review work, and debugging of AI-generated code that passed initial review but failed under conditions nobody anticipated. The agent produces a solution that works for the stated problem. The developer then spends time verifying that the solution does not violate any of the unstated constraints — whether it breaks something relying on the old behavior, whether it removes a guard that was there for a reason, whether it makes an architectural assumption that conflicts with the system's actual design. That verification work is what consumes the time. It should not be necessary if the agent had been operating with complete context from the start.

Three studies, the same mechanism:

**Step 1 — The agent receives a task with incomplete context.** Not incomplete because the requirements are vague — incomplete because decisions were made before this session, constraints exist outside the codebase, and invariants were established through operational experience that was never written down. The agent does not know the context is incomplete. From its position, the task description and the codebase are the complete specification.

**Step 2 — The agent fills the gaps with locally-valid inferences.** In the absence of information, the agent reasons from what it can see. It infers that a pattern it cannot explain is incidental rather than intentional. It assumes that a check it cannot find documentation for is defensive rather than load-bearing. It treats an architectural decision that looks arbitrary as something it can optimise. Each inference is individually plausible. Each inference is made without knowledge of what it is overriding.

**Step 3 — The agent produces output that is locally correct and globally wrong.** The code compiles. The tests pass. The PR review checks out. The implementation satisfies the stated requirements. It also violates an invariant the agent did not know was an invariant, removes a guard that was there because of an incident three years ago, or makes an assumption about an external system that was true in staging and false in production.

**Step 4 — The failure surfaces at the point furthest from the change.** Not in the unit tests. Not in the integration tests. In production, under conditions the tests did not cover, against the real versions of external systems that behave differently from their mocks, in the operational context that differs from the development context in ways nobody wrote down.

That is the mechanism. Not "agents need more context" in the general sense — specifically: agents are operating in systems designed by people who accumulated context over years that was never made machine-readable.

Four failure modes. Each one precise, each one illustrated with an example from this project's history and an example from production systems this project is not.

**Definition:** The agent cannot distinguish between production and non-production environments. It treats environment-specific resources as interchangeable instances of the same type, because nothing in its context tells it otherwise.

**The mechanism:** Environment discrimination is almost never encoded in the codebase. Configuration values point to different endpoints in different environments, but the agent does not know which endpoint is production unless that information is explicitly provided. Naming conventions — bucket names, database names, service identifiers — are often shared across environments with only a prefix or suffix distinguishing them.

**Project example — Issue #6:** The CI/CD pipeline initially had the YAML step starting mock servers independently of the pytest session fixtures. The agent writing the YAML had no way to know that port 8091 was already bound by the test fixtures. It saw "start a server on port 8091" and started one. The fixture code also started one. Both layers believed they owned the same resource. `OSError: [Errno 98] Address already in use`

. Neither layer was wrong in isolation — they had no shared model of who is authoritative for server lifecycle. Production blindness at the infrastructure level.

**Production-system example:** A company runs its notification service with two environments: `notifications-staging`

and `notifications-prod`

. Both configurations are structurally identical. An agent tasked with "improve the reliability of the notification delivery system" adds retry logic to what it believes is the staging environment — because the configurations are identical and nothing in its context marks one as untouchable. The change goes to production. Customers who opted out of marketing emails receive three copies of a promotional notification. The agent did not modify the opt-out state. It modified the retry behavior of a service that now treats previously-terminal deliveries as retryable. The failure is in what the agent did not know, not in what it did.

**Artifact that prevents it:** Environment discrimination documents — an explicit section in CLAUDE.md or a dedicated `ENVIRONMENTS.md`

that names production resources, describes what distinguishes them from non-production resources, and states what the agent may and may not modify in each environment. The agent cannot infer environment boundaries from code structure alone; the document makes the boundary explicit and machine-readable.

**Definition:** The agent cannot access or reason about decisions that were made before the current session. It re-derives patterns from first principles — sometimes correctly, sometimes differently from how they were originally decided — without knowing it is making a decision that was already made.

**The mechanism:** Decisions accumulate in systems in non-queryable forms: in the heads of the people who made them, in commit messages that say "revert X" without explaining why X was wrong, in the post-mortem document that sits in a shared drive nobody opens. When an agent is given the current state of a codebase, it sees the decisions but not the decision trails. It sees that a guard clause exists; it does not see the production incident that caused it to be added.

**Project example — Issues #2, #3, #8:** The step `And the response is returned within 12 seconds`

was introduced in Issue #2. Specific enough to pass. But no anchor: 12 seconds measured from when? The client sends the request? The last retry fires? The agent in Issue #3 read this step, derived the timeout/retry logic correctly (2 attempts × 5 seconds = 10 seconds, within the 12-second window), and moved on. The ambiguity was inherited silently across three sessions before Issue #8's spec audit caught it. The spec had been passing its tests for three issues while carrying an unanchored measurement that two different agents would have implemented differently.

**Production-system example:** A payment service has a 30-second idempotency window on charge requests — added three years ago after a production incident where a dropped TCP connection caused a customer to be charged twice. The window is enforced by a check in the charge handler. An agent tasked with "reduce latency in the payment flow" identifies the idempotency check as overhead for the 99% of requests that are first-time charges. It refactors the check to run asynchronously after the response. For first-time charges, latency drops. For retried charges within the window, the async check completes after the second charge has already been processed. The incident recurs. The idempotency window looked like an optimisation target. It was a load-bearing guard. The agent had no access to the incident report that explained why it was built the way it was.

**Artifact that prevents it:** Architecture Decision Records with agent-readable invariant sections — documents that capture not just what was decided but why, with an explicit section stating the consequences of reversing the decision. ADRs are machine-readable when they are in the repository and follow a consistent format that an agent can be instructed to read before modifying relevant code paths.

**Definition:** The agent does not know which external systems it is affecting, what their failure modes are, or which downstream consumers depend on its output. It acts on the system in view and is blind to the systems connected to it.

**The mechanism:** External dependencies are present in a codebase as HTTP clients, database connections, and API calls — the code is visible. What is not visible is the contract between the service and its consumers: which fields they depend on, what response times they have SLAs against, what error codes they have special handling for. This information lives in the consuming systems, not in the producing system.

**Project example — Issue #7:** The notification service is fire-and-forget specifically because coupling order confirmation to notification delivery is a failure mode, not a feature. An agent asked to "ensure notifications are delivered reliably" might observe that the notification call runs in a daemon thread with no success verification and make the call synchronous. From the agent's perspective, this is strictly more reliable. It also breaks the isolation boundary that was the entire point of the fire-and-forget design. The order service now fails or times out whenever the notification service has a problem. The agent correctly implemented "more reliable notifications" while breaking "reliable order confirmation" — because it had no context about which dependency was supposed to be isolated from which.

**Production-system example:** A payment webhook handler is designed without retry logic. Deliberate: the payment provider already retries webhook delivery up to 25 times over 72 hours. Adding application-level retries would mean that when the provider retries delivery on any 5xx response, the handler processes the same webhook twice. For webhooks that trigger financial operations — refund initiation, commission calculation — duplicate processing means duplicate actions. An agent tasked with "add retry logic to the webhook handler to improve resilience" adds it. The provider retries. The handler retries. A refund is initiated twice. The absence of retry logic was intentional. Nothing in the handler's code explained why.

**Artifact that prevents it:** Dependency maps and external service contracts in CLAUDE.md or a dedicated architecture document — explicit statements of which systems this service calls, which systems call it, what those callers depend on, and which design decisions were made specifically to manage a dependency's failure modes.

**Definition:** The agent does not know which properties of the system must remain true across all changes — the invariants that were never written down because they seemed obvious to the people who designed the system.

**The mechanism:** Some invariants are enforced by tests. Some by types. The invariants that cause the most damage are enforced by convention and institutional memory — the ones that "everyone knows" and nobody documented, because at the time they were established, there was no reason to imagine that anyone would not know them.

**Project example — Issue #2:** The order service always checks inventory before calling the payment gateway. This decision is documented in the Gherkin spec: Scenario 3 states that the payment gateway is never called for out-of-stock items. But "never called for out-of-stock items" does not imply "inventory must be checked first." It only implies that if inventory fails, payment must not be called. An agent asked to "optimise the order flow" might reason that checking payment first is faster for the common case — inventory can be checked in parallel or after. Payment-first, then inventory. Both checks still happen. All five scenarios still pass. Scenario 3 passes because the inventory check still happens. But the system now charges customers before confirming that their items are available. The spec constrained the output. It did not constrain the implementation structure that produces it.

**Production-system example:** A financial transaction flow runs a fraud check before a fund capture. Standard practice. The fraud check becomes a latency bottleneck — a third-party service that has slowed. An agent tasked with "optimise financial transaction latency" proposes running the fraud check in parallel with the fund capture, or after it completes. The agent is not removing the fraud check — it still runs. The tests still pass — both steps complete. But funds are now captured before fraud is detected. When the fraud check returns a positive after capture, the transaction must be reversed. Chargebacks increase. The test suite encoded "fraud check runs" and "capture runs" as separate assertions. It did not encode "fraud check must complete before capture." The invariant was in the ordering. The ordering was not tested. The agent had no way to know the ordering was non-negotiable.

**Artifact that prevents it:** Invariant documentation — explicit statements of properties that must remain true across all changes to the system, separate from the behavioral specs that describe what the system does. "Inventory must be checked before payment is attempted" is an invariant. "Out-of-stock items return 409" is a behavioral spec. Both are necessary. Only the spec is currently encoded in the tests.

After twelve issues: four failure modes, four gaps.

**Production blindness:** CLAUDE.md names the ports and services but does not state which resources an agent may modify, which it may only read, and which it must never touch. The CI pipeline is a shared resource. The `pacts/`

directory is a derived artifact. Neither is marked as such.

**Historical amnesia:** Twelve sessions of decisions are documented in findings files as narrative prose. Accessible — but not queryable. An agent asked to "improve timeout behavior" would not automatically read twelve findings files. It would read the current step definition, infer the intent, and act. The fire-and-forget notification decision, the inventory-before-payment ordering, the two-attempt payment retry cap — none are in a format that says "this is a decision that must not be reversed without reviewing the reasoning."

**Dependency ignorance:** The Pact consumer tests encode which fields the order service depends on from the payment gateway. The reason those specific fields were chosen is not documented. An agent asked to "simplify the payment gateway integration" might remove `transaction_id`

from the response contract because it is not used in any visible business logic.

**Invariant blindness:** The highest-risk gap. The behavioral invariants are in the Gherkin spec. The implementation invariants — properties of the implementation that must survive all future changes — are not encoded anywhere. The inventory-before-payment ordering is visible in `app/main.py`

but not stated as "this ordering must not change." The fire-and-forget notification is implemented in code but not stated as "making this synchronous would violate a design constraint."

This project has twelve sessions of documented history and six active skills. Neither addresses this class of failures. The findings files are too narrative to be queryable. The skills govern how agents produce output; they do not govern what agents must not change. The Gherkin spec constrains behavior; it does not constrain implementation structure. An agent starting Issue #15 with access to all of this infrastructure could still reorder the inventory and payment calls, make the notification call synchronous, or remove a Pact field that looks unused — not because the infrastructure is inadequate for what it was designed to do, but because none of it was designed to answer the question "what must not change?"

In Issue #3, the agent was given only the Gherkin feature file and asked to build the order service from scratch. It derived the complete API contract from plain-language scenarios. It found a portability bug the human author had missed. These are genuinely impressive capabilities. The spec did its job. The agent did its job. The output was correct.

Now consider what would have happened if the task had been slightly different. Not "build the order service from this spec" but "optimise the order service." Same agent, same codebase. The agent reads `app/main.py`

. It sees the inventory check before the payment call. It reasons: for the common case, items are in stock — checking payment first would be faster, and inventory could be checked in parallel. That is a reasonable inference. It also violates the design intent. And the agent would not know it violated the design intent, because the design intent was never written as a constraint on implementation structure.

Or Issue #7. The notification service is fire-and-forget because coupling order confirmation to notification delivery is a failure mode. An agent asked to "ensure notifications are delivered reliably" might make the call synchronous — because synchronous delivery is strictly more reliable. The spec does not forbid this. The tests do not catch it. The skills say nothing about it. The only thing that prevents it is a document that states: the notification call must remain asynchronous; here is why.

The spec is a contract for output. The skill is a contract for process. Layer 3 is the contract for the past — the accumulated weight of decisions made by people who are no longer in the session, for reasons that seemed obvious at the time, in a system that now depends on them in ways nobody wrote down.

Twelve issues in, this project is ready to confront what it cannot prevent. That is what Layer 3 is for.

*Next issue: The CLAUDE.md That Actually Works — side-by-side comparison of three versions for the same project, naive to production-grade, and the five things a production CLAUDE.md must contain.*

**Sources & Further Reading**

*This article was written with the assistance of AI tools.*