The Full Stack: Specs + Skills + Stewardship in a Single System

A developer building an AI-powered software factory integrated three layers—specs, skills, and evals—into a single system to build an order cancellation feature. The system caught three specification gaps before implementation, demonstrating the value of combining all layers. The developer credits Dan Shapiro and Nate B. Jones for the conceptual frameworks.

Building the AI Dark Factory — Issue 19 I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word. Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy-autocomplete-to-the-software-factory Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding Why Most of You Won’t Make It Past Level 2 ” is what triggered this entire newsletter. natebjones.com https://www.natebjones.com/ — Watch the video https://youtu.be/bDcgHzCBgmQ This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 in context of the title of this newsletter on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair. Seventeen issues of building. One question left to answer: does it work? Not in isolation — each layer has been demonstrated in isolation. Layer 1's spec caught the payment field rename in Issue 4. Layer 2's skill review caught the routing signal length in Issue 12. Layer 3's eval caught the synchronous notification change in Issue 17. But those were individual demonstrations of individual artifacts. This issue builds a complete new feature — order cancellation — using all three layers simultaneously, and compares the result to Issue 3, when the agent had only a spec and nothing else. DELETE /orders/{order id} with five business rules: Five rules. Three external service interactions. Two ADRs directly applicable. One new feature that touches every layer of the infrastructure. Before any code, six Gherkin scenarios were written for order cancellation.feature . Then the Gherkin quality skill v2.0 was applied to each one. The skill caught three items the first draft left open. Scenario 1 happy path : First draft — Then the order status is "CANCELLED" . Skill correction: missing HTTP status assertion and missing inventory release service call assertion. The scenario described the outcome but not the observable mechanism. Rewritten to: Then the response HTTP status is 200 , And the response body contains "status" equal to "CANCELLED" , and And the inventory service receives a release request for SHOE-RED-42 and BELT-BRN-M . Scenario 5 idempotency : First draft — Then the response indicates the order is already cancelled . Skill correction: UNDEFINED TERM — "indicates" is not an observable check. Rewritten to: Then the response HTTP status is 200 and And the response body contains "status" equal to "CANCELLED" . No error. No new side effects. Two assertions that two agents would implement identically. Scenario 6 non-cancellable state : First draft — Given an order in PAYMENT PENDING status . Skill correction: UNDERSPECIFIED GIVEN — "an order in PAYMENT PENDING status" does not specify how it got there. Rewritten to: Given an order was created via POST /orders with order ID "order-del-pend-001" and payment is pending . Three corrections before a single line of implementation. Each correction is a decision that would have been made silently in the implementation if the spec had been handed over uncorrected. Three skills were consulted before implementation began. Gherkin quality skill v2.0 — applied to all six scenarios as described above. Three corrections produced. Step definition style skill — consulted before writing tests/steps/test order cancellation.py . The skill's five conventions were followed: fixture injection from conftest.py , mock server state asserted via call log rather than response body, the reset all logs autouse fixture, time.sleep 0.3 before async side-effect assertions, and delete order as the shared helper following the post order naming convention from the existing files. One structural consistency check from the skill: the new step definition file uses def response user id, ... fixture injection, not def test cancellation ... test function style. Without the skill, the file might have used a different fixture pattern — the other test files use this pattern consistently, but an agent reading only order cancellation.feature would not necessarily infer it. Feature file audit skill — run against order cancellation.feature after the quality skill corrections. Found zero additional debt items. The skill's Q5 check "what does this scenario NOT say that it should?" flagged the absence of a scenario for concurrent cancellation attempts — two requests for the same order arriving simultaneously. Documented as a known gap rather than a spec debt item: this is a judgment call about scope, not an ambiguity about behavior. One skill gap revealed: no skill exists for WireMock stub design. The decision to add POST /inventory/release/{scenario} as the release mechanism — rather than a DELETE or a patch to the existing inventory endpoint — was made without consulting any artifact. This is an implicit decision that a stub design skill would have caught. Before any modification to app/main.py , the operation scope eval was run. Q1 — ADR coverage: The decision index identified two applicable ADRs. ADR-001 agent check questions applied to the cancellation flow: ADR-002 agent check questions applied to the notification call: Both ADR checks passed before implementation began. The notification thread was implemented correctly on the first attempt — not because the agent inferred it, but because ADR-002 stated explicitly that asynchronous was non-negotiable and described the daemon thread pattern. Q2 — Ordering of external service calls: Cancellation flow ordering documented before implementation: inventory release first, then notification fire-and-forget . Payment gateway not called. Ordering committed in the findings before the first line of code. Q3 — Synchronicity: The notification call — async. The inventory release call — synchronous. Cancellation confirmation depends on inventory release success; it does not depend on notification delivery. The environment eval was not triggered — no infrastructure files were modified. DELETE /orders/{order id} implemented in app/main.py . The cancellation logic: python @app.delete "/orders/{order id}" def cancel order order id: str : order = ORDERS.get order id if not order: raise HTTPException status code=404, detail=f"Order {order id} not found" Idempotency: already cancelled returns success if order "status" == "CANCELLED": return {"order id": order id, "status": "CANCELLED", "message": "Order already cancelled.", "status code": 200} Only CONFIRMED orders can be cancelled if order "status" = "CONFIRMED": return {"order id": order id, "status": order "status" , "message": f"Order cannot be cancelled. Current status: {order 'status' }", "status code": 422} Release inventory reservation — synchronous, cancellation depends on it try: inv resp = httpx.post f"{INVENTORY URL}/inventory/release/success", json={"order id": order id, "items": i "sku" for i in order "items" }, timeout=5.0 if inv resp.status code = 200: raise HTTPException status code=503, detail="Inventory release failed" except httpx.RequestError as e: raise HTTPException status code=503, detail=f"Inventory service unavailable: {e}" Update order status order "status" = "CANCELLED" order "cancelled at" = datetime.utcnow .isoformat Notify — fire-and-forget per ADR-002 def notify : try: httpx.post f"{NOTIFICATION URL}/notifications/order-cancelled", json={"order id": order id, "user id": order "user id" }, timeout=5.0 except Exception: pass Notification failure does not affect cancellation threading.Thread target= notify, daemon=True .start return {"order id": order id, "status": "CANCELLED", "message": "Order successfully cancelled.", "status code": 200} Test results after first implementation attempt: pytest tests/steps/test order cancellation.py -v test successful cancellation of confirmed order PASSED test idempotent cancellation of already cancelled order PASSED test cancellation of non existent order PASSED test rejection of cancellation for payment pending order PASSED test inventory release on cancellation PASSED test fire and forget notification on cancellation PASSED 6 passed in 4.23s 6/6 on the first attempt. Full suite: pytest tests/steps/ -v → 21 passed pytest tests/pact/ -v → 2 passed python scripts/can i deploy.py → SAFE TO DEPLOY Issue 3 was the first agent implementation session. The agent was handed the Gherkin scenarios for the order creation endpoint and told to build. It derived the entire API contract correctly and found a portability bug in the human's code. It also made four implicit decisions. Here is the comparison across five dimensions. Implicit decisions made Issue 3: 4 implicit decisions — HTTP 404 for missing orders, ISO string timestamp format, in-memory store, no failure scenario for partial availability. Issue 19: 1 implicit decision — the POST /inventory/release/success URL path convention for the release stub. The stub design skill does not exist; this decision was made without consulting any artifact. The three layers reduced implicit decisions from 4 to 1. The remaining one identifies a gap in the skill infrastructure, not a gap in the implementation. Spec quality before implementation Issue 3: the spec was handed to the agent without review. The timeout ambiguity And the response is returned within 12 seconds was introduced in Issue 2 and silently inherited for three sessions before Issue 8 caught it. Issue 19: the Gherkin quality skill caught three items before implementation. The idempotency scenario's "indicates the order is already cancelled" would have produced an ambiguous assertion in the step definition — the agent would have invented a field name. The skill replaced it with two concrete assertions before the step definition was written. Stewardship artifacts consulted Issue 3: none existed. Issue 19: three artifacts prevented three potential failures. ADR-002 prevented the notification call from being made synchronous. Without it, the agent's default for a "confirm delivery" side effect would have been to await the response. ADR-002 stated the daemon thread pattern explicitly; the implementation used it on the first attempt. ADR-001's Q2 check prevented an inventory-before-payment ambiguity in the cancellation flow — not because payment was involved, but because the ordering question was asked before implementation and documented. The inventory release was confirmed as the primary side effect before a line of code was written. The operation scope eval's Q3 caught the notification synchronicity risk before implementation began. Not after a test failure — before the function was written. Time spent on clarification vs implementation Issue 3: clarification was zero — the agent inferred everything. Implementation took one pass. Four implicit decisions were made invisibly. Issue 19: pre-flight work — spec review, eval, ADR checks — took approximately the same time as the implementation itself. The implementation took one pass. One implicit decision was made, and it was documented in the findings rather than embedded silently in the code. The overhead is real. The pre-flight is not faster than inferring. What it produces is different: documented decisions rather than invisible ones. Test results after first implementation attempt Issue 3: 5/5 passed — an exceptional result that Issue 3's article noted was partly due to the simplicity of the API. Issue 19: 6/6 passed — on an endpoint with more complexity idempotency, two side effects, state machine , tested against six scenarios that the Gherkin quality skill had already hardened. One implicit decision made in this session that none of the three layers caught: the stub URL path convention for inventory release. The decision: POST /inventory/release/{scenario} mirrors the existing stub pattern of POST /inventory/check/{scenario} . This is the right choice — it is consistent with the project's conventions. But it was made without consulting any artifact. A stub design skill that documented the URL convention for mock endpoints would have made this explicit rather than inferred. This is the only true gap. Everything else was either covered by an artifact or was a judgment call that should remain with the human: which scenarios to write scope , whether to add a concurrent cancellation scenario product decision , what error message text to use editorial . The distinction that matters: some judgment calls are implicit decisions that should be documented. Others are genuinely human judgment that no artifact can or should replace. The stub URL convention is the first category — it has a right answer that the project has already established. The concurrent cancellation scenario is the second category — it is a product decision about scope, not an engineering ambiguity about behavior. After seventeen issues of infrastructure, the remaining gap is small, specific, and nameable. That is the point. Issue 3, but harder. The agent would have built a working cancellation endpoint. It would have made the notification call synchronous — because synchronous is the natural default for a side effect you want to confirm. It would have written Then the order is cancelled in the Gherkin scenarios — which would have produced an ambiguous step definition. It would have made the inventory release a flag rather than a service call — because inventory released: true in the response body is simpler than an HTTP call to a separate service. All three tests would have passed. The notification synchronicity violation would not have been caught until the first 2am notification incident. The spec ambiguity would have been inherited by the next agent session. The inventory flag would have stayed a flag until someone noticed it was not actually releasing inventory. The three layers did not produce a faster first implementation. They produced a better one — with fewer invisible decisions, and one documented gap that the next session can close. Next issue: The Productivity J-Curve — the honest accounting of what seventeen issues of infrastructure building actually cost, and whether the answer to "is it worth it" is yes. Sources & Further Reading This article was written with the assistance of AI tools.