I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.
Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com
Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video
This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.
Before Issue #9, this project had one reusable quality check: a pasted prompt. No routing signal. No output contract. No version. No idempotency guarantee. The prompt worked because the human author remembered to paste it, and because the agent happened to be in the right context to apply it.
That is not infrastructure. That is memory.
Layer 2 ends here. Five issues of building skills, organising them into tiers, stress-testing them, reviewing them. This issue does the stocktake: a full inventory of everything that accumulated across twelve issues, classified against the tier model, with a conversion decision for each item.
The uncomfortable finding: after twelve issues of careful, documented work, there was more living in memory than anyone expected.
Searching every source — CLAUDE.md, all skill files, the prompt directory, the step definition files, the patterns implicit in twelve sessions of findings — produced seventeen items. Here's what the classification revealed:
Already properly converted to skills: 4
The Tier 1 formatting standard, the Gherkin quality skill v2.0, the output formatting standard, the session findings protocol (partially). These are infrastructure. They have routing signals, output contracts, and versions.
HIGH priority — unconverted and dangerous: 4
The session start and documentation protocol. Lives as prose in CLAUDE.md. Every session requires it. No routing signal, no output contract. An agent follows the prose or guesses. The pattern has been consistent across twelve sessions not because a skill enforces it — because the same author pasted the same instructions each time.
The spec-audit-framework.md. Contains the complete five-question diagnostic, the six-class debt taxonomy, the fix rubric, and the scorecard template — everything needed for a Tier 2 skill. What it's missing: a routing signal, a when-to-use section, an output contract. It's the methodology without the interface.
The step definition writing pattern. Five conventions followed in every test file across twelve issues. Fixture injection from conftest.py
. Mock server state asserted via call log, not response body. Async side effects with time.sleep(0.3)
before assertion. The reset_all_logs
autouse fixture for cleanup. Helper extraction with _post_order
-style naming. Not one of these is written down anywhere except in the code itself.
The original Gherkin quality prompt (v1.1 skill). Both the docs/prompts/
version and the docs/skills/tier2/gherkin-scenario-quality.md
v1.1 are active liabilities. An agent finding either gets the pre-guard version — no idempotency protection, no domain check, the four failure modes that Issue #11's stress tests found. Both need to be deprecated before they're found instead of the v2.0 skill.
Items that don't fit any tier: 1
The newsletter audience description in CLAUDE.md. Not an agent instruction — project metadata for tone calibration. Correctly stays as context, not converted.
The three highest-priority unconverted items were converted to proper skills in this session, each self-reviewed against the Issue #12 five-dimension checklist before committing.
Feature file audit skill (Tier 2) The spec-audit-framework.md methodology finally has a callable interface. The new skill routes correctly — "Audit order_creation.feature for spec debt" routes to it; "Fix this scenario's underspecified step" routes to the Gherkin quality skill instead. The two skills now form a pair: the audit skill identifies debt, the quality skill fixes it.
Self-review verdict: APPROVED. Step definition style skill (Tier 2)
Twelve issues of implicit convention, now explicit. The five patterns are documented with reasoning — not just "do this" but "do this because the conftest fixture-injection pattern prevents port conflicts when running pytest tests/steps/
across multiple step files simultaneously."
One open finding from the self-review: helper function naming and extraction is under-specified. An agent may or may not extract _post_order
-style shared helpers. The test files will work either way. They will not be structurally consistent. Flagged for v1.1.
Self-review verdict: APPROVED WITH COMMENTS.
Session start protocol skill (Tier 1)
The documentation protocol prose in CLAUDE.md, converted to a Tier 1 skill with a routing signal and an explicit output contract covering when to create the findings file, what the file must contain before any implementation begins, and what "write in real time" means operationally.
Self-review verdict: APPROVED. Two items deprecated in this session. Neither was a surprise — both had been carrying liability since Issue #11 identified the failure modes they contain.
The original prompt (docs/prompts/prompt-gherkin-scenario-quality.md
). The "before" artifact from Issue #9. An agent finding this file gets the pre-skill version of the quality check — no output contract, no guards, no idempotency protection. Issue #11 found four failure modes in the v1.1 skill; the prompt has all four plus additional divergence risk from the missing output contract.
The v1.1 skill (docs/skills/tier2/gherkin-scenario-quality.md
). Received a CHANGES REQUESTED verdict in Issue #12's review. The v2.0 skill header says "Supersedes v1.1" — but an agent that reads v1.1 directly has no way to know it's deprecated. The deprecation header is now explicit.
The principle both items illustrate: a deprecated artifact that is still discoverable is not deprecated. It is a trap. An agent routing for "Gherkin quality check" in this repo could find either the prompt (pre-skill, pre-guard), the v1.1 skill (UNSTABLE idempotency, PLAUSIBLE WRONG for UI scenarios), or the v2.0 skill (four guards, APPROVED). All three are in the same filesystem. Only one is the right answer.
The standalone deliverable for this issue lives at docs/skill-audit-template.md
in the repo. It works without reading this newsletter. Here's the core of it:
What counts as a prompt library item: prompts you paste into sessions, system prompts or CLAUDE.md-style config files, reusable instructions embedded in larger documents, implicit conventions derivable from existing outputs, and existing skill files.
The five conversion questions:
The junk drawer test — four questions that find what should be deprecated:
The full template with classification taxonomy, conversion checklist, and audit scorecard is in the repo.
After converting three items and deprecating two, the inventory is cleaner. It is not clean.
The implementation layer is undocumented. Every implementation session relies on an agent reading app/main.py
and inferring the patterns: the FastAPI route structure, the in-memory order store, the mock-stub selection via request body, the background notification thread. These conventions are consistent across twelve issues — consistent because the same files were read each time, not because a skill exists to enforce them.
The why-this-matters
skill remains at Tier 3. The Issue #10 promotion decision documented three criteria that must be met before it becomes Tier 2. None were met in this session. An agent writing findings for a new session does not automatically find this skill.
The step definition style skill has an open finding. Helper function extraction is under-specified. The test files work. They may not be structurally consistent across future sessions.
The honest answer to "what's still living in someone's head after twelve issues of careful work?" is: the reasoning behind app/main.py
's design decisions. Why the mock stub selection happens via request body parameters rather than URL routing. Why the notification is fire-and-forget with a 0.3-second sleep rather than synchronous. Why the in-memory order store is designed the way it is. These decisions are documented in the findings files where they were made — Issues #2, #3, #7. They are not in a skill. They are not in CLAUDE.md. An agent implementing a new endpoint reads the code, derives the patterns, and probably gets them right. An agent implementing after a refactor might not.
Layer 3 begins with this inheritance: a working skills infrastructure, a review process, an audit process, three new Tier 2 skills, two deprecated liabilities, one under-specified skill, one promoted-but-not-merged skill, and an undocumented implementation layer that is the highest-risk gap in the project.
Five issues. One prompt to start, converted to a skill. That skill stress-tested, reviewed, and versioned. A tier architecture to organise it. A review framework to evaluate future skills before they ship. An audit process to find what was never documented.
The most dangerous item in the inventory was not the deprecated prompt or the v1.1 skill. It was the five implicit step definition conventions followed in every test file across twelve issues without ever being written down. They worked because they were derived from the same source files each time. They would not have survived a context switch, a project transfer, or a version upgrade — because the standard existed only in the consistency of the inputs, not in any document.
Auditing a prompt library is not about finding what failed. It is about naming what hasn't failed yet and asking how much of that success is documented versus remembered.
After twelve issues of careful work, this project had more in the second category than anyone expected.
Next issue: The Memory Wall — why agents fail in production, the honest numbers, and the taxonomy of failure modes that Layer 3 is built to prevent.
Sources & Further Reading
This article was written with the assistance of AI tools.