I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.
Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com
Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video
This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.
Issue #11 stress-tested the Gherkin quality skill and found four failure modes. Issue #11 then fixed them. The resulting v2.0 skill passed every adversarial input.
This issue asks the question that should have come first: how do you review a skill before the stress tests tell you what's wrong?
The answer matters because stress tests and skill reviews catch different things. A stress test answers "does the skill work when called?" A review answers "is the skill ready to be called in all the contexts its description implies?" They are complementary. Running the stress tests first and the review second — as happened in Issues #11 and #12 — is the wrong order.
Code review is a solved problem. You review the diff. You check the logic. You ask "does this code do what it should?" and you either approve or request changes.
Skill review is not a solved problem. The review target is different. You are not asking whether the code is correct. You are asking:
None of these questions can be answered by reading the code. They can only be answered by working through a structured checklist.
The review framework built in this session covers five dimensions. Every numbered question must be answered before a skill version can be approved. A reviewer who reads a skill and asks "does this look reasonable?" is not doing a skill review.
Dimension 1 — Routing signal
Is the description on a single line and under 120 characters? Does it name the artifact type, the domain scope, and the methodology — specifically enough to route correctly and generally enough not to misroute?
The test: write three prompts that SHOULD route to this skill and three that SHOULD NOT. Verify each. Document any misroutes.
Dimension 2 — Output contract
Is the contract explicit and enumerable — every requirement a yes/no check, not a judgment call? Could two agents produce different outputs that both satisfy it? Does the contract specify what the skill must NOT produce, not just what it must?
The test: identify the downstream consumer. Document what it does with the skill's output. Ask whether the contract is sufficient for that consumption pattern.
Dimension 3 — Methodology
Does the methodology describe reasoning or procedure? Pick three edge case inputs not covered by the methodology examples. Apply the methodology manually. Document whether it produces correct output for each.
The test: identify domain knowledge that an agent cannot infer from first principles. It must be stated explicitly, not implied.
Dimension 4 — Idempotency and stability
Apply the skill to the same input with three different framings. Do all three produce structurally identical output? Apply the skill to an already-correct input. Does it return unchanged or rewrite unnecessarily? Apply the skill to its own output. Does it return unchanged?
Dimension 5 — Failure modes
Test with one out-of-scope input, one contradictory input, one empty input. For each, classify the output: FAIL SIGNAL (explicit failure, no output), PLAUSIBLE WRONG (looks correct, contains error), or CORRECT REFUSAL (actionable error message). Are all PLAUSIBLE WRONG outcomes eliminated?
The v1.1 review is the review that should have happened before the Issue #11 stress tests were needed. Working through all five dimensions found findings the stress tests could never have found — and confirmed exactly which failures the stress tests did find.
Routing signal — 137 characters against a 120-character limit. The description is 17 characters over the threshold above which many agent routing frameworks truncate or deprioritise the signal. The excess carries "and output contract" — meaningful to the skill author, invisible to an agent routing on a 120-character budget. The stress tests in Issue #11 could not find this — they test behaviour when the skill is invoked. They cannot test whether the skill is invoked.
Output contract permits agent divergence. Four under-specified requirements allow two agents to produce different outputs that both satisfy the contract: scenario title "explicit" criterion, required fields per scenario type, required HTTP status codes per outcome, required external services per scenario type. Stress tests verify one agent's output. They cannot reveal the latitude available to a second agent working from the same contract.
Methodology gap for missing Given clause. The Q3 check asks whether terms are defined, but not whether the precondition state is established. A scenario with no Given clause passes Q3 if none of the steps use undefined nouns. The stress tests used well-formed inputs; this gap was not in the input set.
Two PLAUSIBLE WRONG failure modes confirmed:
Both are exactly the failures the Issue #11 stress tests found. A pre-v2.0 review using this checklist would have required explicit termination for both cases — and v2.0's Guards 2 and 3 might have been built before the stress tests were necessary.
v1.1 Review verdict: CHANGES REQUESTED.
The v2.0 review confirms that the four stress-test failures are fixed. It also finds three issues the stress tests missed.
Routing signal is now 179 characters. v2.0 made the signal 42 characters longer than v1.1's already-failing signal. The addition of ", four pre-flight guards, and a minimal-change" describes internal implementation mechanisms that are irrelevant to a caller routing to this skill. The routing signal now describes how the skill works internally rather than what it produces.
Guard 4 return value format is ambiguous between two instructions. This is the most important finding in the session. Documented in full below.
Guard 4 gap for missing Q5 assertions. A scenario that passes all five Guard 4 format conditions — concrete IDs, named services, HTTP status, no UNDERSPECIFIED patterns, field+value Then clauses — but is missing Q5 side-effect assertions (payment gateway call count, inventory reservation assertion) triggers Guard 4 and returns "no changes required." The skill signals completion for an incomplete scenario. The stress tests in Issue #11 did not test this input type because the stress tests focused on the four failure modes v2.0 was designed to fix.
Two new edge cases introduced:
v2.0 Review verdict: APPROVED WITH COMMENTS.
The four major failure modes are addressed. The new issues are not blocking. v2.0 is ready to be the canonical version — with a v2.1 planned.
PR:gherkin-scenario-quality-v2.md
— Agent-safe Gherkin quality skill
Section:Pre-flight guards → Guard 4 (Idempotency check)
Guard 4 has two return instructions that conflict, and the conflict matters at agent scale.
The "Return:" block shows only the annotation comment:
Then the next line says: "If Guard 4 triggers, return the input scenario unchanged."
Together these read as: return the comment block, AND return the input scenario. But a skill returns a single value. The two instructions imply three possible interpretations: (a) the annotation only — the scenario is not in the output; (b) the annotation prepended to the scenario, matching the minimal-fix pattern used elsewhere; or (c) the scenario with the annotation appended.
The rest of the skill uses format (b). Guard 4's "Return:" block uses format (a). The inconsistency is invisible when a human reads the output and manually pastes the scenario into a feature file — the human ignores the comment and pastes the scenario. But in an automated pipeline it is not invisible.
Concrete impact: A downstream agent that receives Guard 4 output and writes all skill output to a feature file would write the # SKILL: No changes required
annotation as a Gherkin comment into the file. At pipeline scale across 50 feature files, that is 50 permanent skill-internal annotations committed to production specs. If the agent uses interpretation (a) and treats the annotation block as the complete output, the original scenario is silently discarded — replaced by two comment lines.
Suggested fix: Align Guard 4's return spec with the minimal-fix annotation pattern:
Return the input scenario with the following comment prepended:
[followed by the complete input scenario, unchanged]
Alternatively, if the annotation is caller metadata and NOT part of the Gherkin output, state this explicitly: "The guard annotation is caller metadata. Do not include it in the feature file. Return it as a separate response block before the unchanged scenario."
Either formulation eliminates the ambiguity. The current text requires the downstream agent to guess.
Why this is the most important finding from either review: v2.0 was built explicitly to be safe at agent scale. The four guards exist because automated pipelines create failure modes that human callers handle silently. Guard 4's return value specification has the same class of failure it was designed to prevent — a human reading the output knows which part is the scenario and which part is metadata; an automated pipeline does not. The stress tests verified that Guard 4 triggers correctly. What they could not verify — because they test the skill in isolation — is whether Guard 4's output is correctly specified for all downstream consumers.
The stress tests in Issue #11 found three behavioral failures and confirmed a fourth. This session's review found three findings the stress tests could not reach:
Stress tests answer "does the skill work?" Review answers "is the skill ready for every context?" The findings that stress tests cannot reach are the ones where the skill is correctly invoked, correctly produces output, and a downstream system still fails — because the output format was not specified for that consumption pattern, or because the routing signal was too long to fire reliably, or because a guard fired on valid input.
Running the stress tests first, as happened in Issue #11, found the acute failures. Running the review second, as happened here, found the ones that would have surfaced later — quietly, in production, without a clear signal that the skill was the cause.
The correct order is review first, stress tests second. The review tells you where to aim the stress tests.
Next issue: The Skill Audit — walking through the full prompt library accumulated across twelve issues, applying the tier framework, and building the audit template readers can use on their own libraries.
Sources & Further Reading
This article was written with the assistance of AI tools.