{"slug": "poka-yoke-mistake-proofing-claude-code-skill-for-software", "title": "Poka-Yoke: Mistake-Proofing Claude Code Skill for Software", "summary": "Poka-yoke, a mistake-proofing skill set for AI coding assistants, improves the rate at which models identify design constraints from 42% to 81%, according to benchmarks from developer rainmanjam. Across 591 blind-graded runs on six runtimes, the skills boosted performance for models including Fable 5 (+8.3 pp), Opus 5 (+3.6 pp), Sonnet 5 (+8.6 pp), and Haiku 4.5 (+12.9 pp), but also reduced detection of specific defects like raw SQL injection from 92% to 69%. The open-source project, inspired by Shigeo Shingo's industrial method, offers a hazard scanner for TypeScript, Python, Go, Rust, and SQL, and 11 skills for design, installable via pre-commit, CI, or lint hooks.", "body_md": "An agent will tell you what to fix. It will rarely tell you what its fix makes impossible.Unprompted, models close a design by naming what it forecloses\n\n42%of the time. With these skills,81%, measured across 80 graded verdicts and six model families. That one habit is most of what this does, and it is the difference between advice you agree with and a constraint you can rely on.A dependency-free\n\nhazard scannerfor TypeScript, Python, Go, Rust and SQL; installablepre-commit / CI / lint / hook devices; and11 skillsthat apply the same method while you design.[Shigeo Shingo]'s method, applied to code, process, interfaces, infrastructure, and AI.Runs on\n\n19 agent runtimesfrom one set of skills, 10 with a native manifest. Benchmarked on Claude Code;[the support tiers are stated honestly].\n\nNote\n\n**591 blind-graded runs across six runtimes**, at the first turn of a fresh session, scored\nagainst pre-written assertions by a grader that never sees which configuration produced a\nresponse. Every model improves on a no-skill baseline by more than noise: Fable 5\n**+8.3 pp**, Opus 5 **+3.6 pp**, Sonnet 5 **+8.6 pp**, Haiku 4.5 **+12.9 pp**: all four 95%\nintervals exclude zero. Nine of the 52 cells came out negative, where chance alone would\nproduce about 18: individual cells hold 1 to 7 runs and are too small to read on their own,\nso the scarcity of regressions is the signal rather than their existence.\n[See the numbers](#benchmarks).\n\nAll 591 runs are verified against the scenario prompts as they stand in this commit.\n\n**What this does not yet establish.** Blind grading controls bias, not accuracy. The baseline\nis *no skill*, not *a different methodology*, so whether any structured method in context\nwould do the same is untested; a control arm is designed and unrun. And every run is a first\nturn, which measures the ceiling rather than what survives an afternoon of accumulated\ncontext. [The full list of what the numbers cannot tell you](#caveats).\n\n[What it trades](#what-it-trades), read this before installing[Install](#install)·[Requirements](#requirements)·[Other runtimes](/rainmanjam/poka-yoke/blob/main/docs/install.md)·[Updating and uninstalling](#updating-and-uninstalling)[The hazard detector](#the-hazard-detector), the part that does not decay[What's inside](#whats-inside), 11 skills, starting with`design`\n\n[The method](#the-method): the two axes that do the work[What it looks like](#what-it-looks-like)·[Invocation](#invocation)[Benchmarks](#benchmarks)·[Caveats](#caveats)·[Repo layout](#repo-layout)·[Prior art](#prior-art)[Contributing](/rainmanjam/poka-yoke/blob/main/CONTRIBUTING.md)·[Releasing](/rainmanjam/poka-yoke/blob/main/RELEASING.md)·[Changelog](/rainmanjam/poka-yoke/blob/main/CHANGELOG.md)·[Code of conduct](/rainmanjam/poka-yoke/blob/main/CODE_OF_CONDUCT.md)·[License](#license)\n\nLoading a method changes what a model attends to, and attention is finite. Measured across the same 591 runs, the skills make responses markedly more constructive and slightly worse at noticing the specific defect already in front of them:\n\n| Behaviour | Baseline | With skill |\n|---|---|---|\n| Proposes a concrete device per finding, not \"add validation\" | 62% | 100% |\n| Names what the design makes impossible | 42% | 81% |\nNotes `pre-commit` is bypassable and must be backed by CI |\n35% | 92% |\n| Identifies a raw SQL interpolation as an injection vector | 92% | 69% |\n| Explains why a silently wrong number beats a failed pipeline | 54% | 31% |\n\n**So: if you want the bug in front of you found, use a reviewer. If you want the shape changed\nso that class of bug stops being expressible, use this.** Loading it for the first job makes\nthe model measurably worse at it. That trade is the product, not a caveat about it.\n\n**People will always make mistakes. That is not the problem worth solving: the problem is\nletting a mistake become a defect.**\n\nShigeo Shingo: the industrial engineer who named poka-yoke and worked it out on the factory floor, built his quality method on that distinction. Instead of asking workers to be more careful, he redesigned the work so a mistake could not survive it. Assemblers kept forgetting a spring inside a switch, so he had them lay both springs in a dish first: a spring still in the dish was the error announcing itself, before the unit could move on.\n\nThis plugin applies that method to software. Not as a metaphor: the taxonomy is the actual\nworking tool. Every finding is classified by **what happens when the mistake occurs** and\n**how the device notices**, which is what keeps it from collapsing into generic code review.\n\n**The line that does most of the work:**\n\nA comment, a docstring, a wiki page, a review checklist, or a line in CLAUDE.md saying \"don't do X\" is\n\nnota poka-yoke. It is training, and training degrades. A device does not. If your fix relies on someone remembering something, keep going.\n\n**You have already tried writing the rule down.** Every repository that needs this has a\n`CLAUDE.md`\n\n, an `AGENTS.md`\n\nor a `CONTRIBUTING.md`\n\nwith a list of things everyone is supposed to\nremember, and the list is longer than it was a year ago because the reminders did not hold. That\nlist is the real alternative to this plugin, not an unassisted model, and the argument here is\nabout why it decays and what to put in its place.\n\nThat is also the honest limit of the current evidence: the benchmark compares *skill* against\n*no skill*, which is not the comparison you face. The comparison you face is against the rules\nfile you already wrote, and it has not been run yet.\n\n```\n/plugin marketplace add rainmanjam/poka-yoke\n/plugin install poka-yoke@poka-yoke\n```\n\nInvoke the mode you want directly:\n\n```\n/poka-yoke:audit      # or -design, -retro, -ops, -authz, -ux, -data, -llm,\n                                # -guardrails, -agent-guardrails\n```\n\nOr just ask for it by name, \"poka-yoke this repo\", \"mistake-proof this API\", \"run a poka-yoke audit on src/billing\". Both work.\n\nWhat does **not** currently work is Claude reaching for these unprompted from a plain\ndescription of a problem. That's measured, not assumed, see\n[invocation](#invocation).\n\n**Without the marketplace**: copy the skills in directly. Use `.claude/skills/`\n\nrather than\n`~/.claude/skills/`\n\nto commit it for the whole team. The second `cp`\n\nlands beside `skills/`\n\n,\nnot inside it, because every SKILL.md reaches its references and scripts at `../../`\n\n:\n\n```\ngit clone https://github.com/rainmanjam/poka-yoke /tmp/pk\nmkdir -p ~/.claude/skills\ncp -r /tmp/pk/plugins/poka-yoke/skills/* ~/.claude/skills/\ncp -r /tmp/pk/plugins/poka-yoke/{references,scripts,assets} ~/.claude/\n```\n\n**Other runtimes**: poka-yoke ships native manifests for Codex, Cursor, Devin, Kimi, Hermes,\nGemini CLI, Grok, Qoder and Kiro; pointer files for Copilot, Windsurf, Cline, Junie, Zed,\nAider and Antigravity; and vendoring instructions for opencode and Pi. Codex, Copilot CLI and Gemini CLI can share one install via `~/.agents/skills/`\n\n.\nSee ** docs/install.md**, which states the support tiers honestly:\n\n**Claude Code, Codex and Antigravity are benchmarked**, and the other manifests are verified structurally in CI rather than behaviourally on the runtime.\n\n**Nothing** for the skills themselves. They are plain Markdown with relative references, which is what lets them load on 19 runtimes rather than one.**Python 3.9+** for the detector. Standard library only; no dependencies to install, so no dependency supply chain.**git** for diff-aware scanning (`--diff`\n\n,`--staged`\n\n,`--since`\n\n). Without it, use`--paths`\n\n.\n\n```\n/plugin marketplace update poka-yoke\n/plugin uninstall poka-yoke@poka-yoke\n```\n\nThe plugin ships a dependency-free scanner for textually-detectable hazards across TypeScript, Python, Go, Rust, and SQL.\n\n**Try it on your own code without installing anything.** No plugin, no marketplace, no agent:\n\n```\ngit clone --depth 1 https://github.com/rainmanjam/poka-yoke.git /tmp/poka-yoke\npython3 /tmp/poka-yoke/plugins/poka-yoke/scripts/cli.py detect --paths /absolute/path/to/your/repo\n```\n\nThe second path must be absolute, or it resolves against the clone rather than your project. Standard library only, so there is nothing to install and nothing to uninstall.\n\nOnce the plugin is installed, the same scanner is available in-repo:\n\n```\npython3 plugins/poka-yoke/scripts/cli.py detect --diff              # changed lines only\npython3 plugins/poka-yoke/scripts/cli.py detect --paths src/ --json\npython3 plugins/poka-yoke/scripts/cli.py detect --severity high\n```\n\nSkills reference it by a path relative to the SKILL.md that names it, so it resolves on any runtime where the plugin directory was copied as a unit: no plugin-root variable, and no package registry.\n\nIt finds adjacent same-type parameters (via real AST parsing for Python), swallowed errors, unbounded deletes, durations with no unit, money as a float, unvalidated parses, and retryable effects with no idempotency key, each tagged with its catalog ID, its lens, and the device that closes it.\n\n**The three numbers, and what each counts.** They measure different things and are easy to\nconflate:\n\n| Count | What it is | |\n|---|---|---|\n| Catalogued hazard shapes | 28 |\nThe taxonomy in\n`references/hazard-catalog.md` |\n\n**20**`C1`\n\n, `F3`\n\n) come from AST checks rather than the pattern table, which is why counting `RULES`\n\nalone gives 18.**42****19**`--all`\n\nruns them anyway.**Scope.** It detects hazards that are visible in the text and surfaces the review question\nbehind each one. Semantic and interface-design judgements are not textual and stay with a\nhuman, or with the skills. Expect real false positives on the pattern rules; that is the price\nof a first pass that needs no configuration and no install.\n\nEleven skills. **Start with design**: it is the one you reach for while building, and\nmistake-proofing is cheapest before the code has callers; every other mode is cleanup by\ncomparison. Its measured effect is uneven:\n\n**81% → 100%** on Fable 5,\n\n**92% → 96%** on Opus 5, and flat on Sonnet 5 and Haiku 4.5. The largest gains in the suite are elsewhere,\n\n`ops`\n\nand `build-endpoint`\n\non Haiku 4.5, so take this as a recommendation about *when*mistake-proofing pays, not a claim that this skill benchmarks best.\n\n```\n/poka-yoke:design      # you're about to write it; make misuse unrepresentable\n```\n\nThe rest, roughly in the order you meet them across a feature's life:\n\n| Skill | Reach for it when |\n|---|---|\n`design` |\nWriting an API, schema, or state model, the hero; start here |\n`poka-yoke` |\nAnything else, applies the method directly and routes when a mode fits |\n`ux` |\nBuilding a form, a destructive action, a flow users can get wrong |\n`authz` |\nAdding anything multi-tenant, permissioned, or IDOR-shaped |\n`llm` |\nShipping an AI feature, structured output, tool gates, evals |\n`guardrails` |\nMaking a rule stick: pre-commit, CI, lint, database constraints |\n`ops` |\nDeploying, migrating, changing infrastructure |\n`data` |\nPipelines and metrics, where failure is silently wrong numbers |\n`agent-guardrails` |\nConstraining an AI agent working on your repo |\n`audit` |\nCode that already exists, find the footguns, rank by damage |\n`retro` |\nSomething broke, kill the whole class, not the instance |\n\nEach mode carries the full method for its domain, so only the one you need is ever loaded.\n\nA strict preference ladder. Always reach for the highest rung you can afford.\n\n| Rung | Software | |\n|---|---|---|\n1 |\nControl: the mistake is impossible |\nType won't compile · `NOT NULL` / `CHECK` · required argument · PreToolUse deny · branch protection |\n2 |\nWarning: possible, but announced as it happens |\nLint error · failing CI gate · runtime assertion · confirmation naming the exact object |\n3 |\nDetection: it ships, something finds it later |\nTests · monitoring · reconciliation |\n0 |\nnot a poka-yoke |\nDocs · comments · training · \"be careful\" |\n\nShingo's three detection methods, mapped to code. These are inspection **lenses**: run all\nthree over an interface and you find hazards a general review misses.\n\n| Method | Factory | Ask code | Devices |\n|---|---|---|---|\nContact |\nthe part won't seat unless correctly shaped | Can the wrong thing fit? |\ndistinct types · branded IDs · parse-don't-validate · units in the type · discriminated unions |\nFixed-value |\na counter confirms all 6 screws | Can a wrong count or incomplete set pass? |\nexhaustive `match` · required fields · row-count guards · config validated at boot |\nMotion-step |\na sensor confirms step 3 before step 4 | Can the steps happen out of order? |\ntypestate · builders · state machines · idempotency keys · RAII / `defer` |\n\n**Source inspection**: check the*conditions*before the error. Designed in where you can, enforced at runtime where you cannot. Best.**Self-check**: the work checks itself. Runtime. Fail fast.** Successive check**: the next station checks. Review, CI.\n\nA CI gate that catches a bad migration is good. A schema that makes it unwritable is better, and costs less forever.\n\nAsk for an audit and you get findings classified, not opinions listed:\n\n```\n### 1. Account IDs can be swapped in transfer(): Money movement / Silent\nWhere:  src/payments/transfer.ts:42\nMistake: Call transfer(dst, src) with the accounts reversed\nConsequence: Funds move the wrong way. Compiles, passes review, silent at runtime.\nToday:  None\nDevice: Brand AccountId as SourceAccount / DestinationAccount → Control\n```\n\nEvery finding names the **mistake**, never the mistaken. Not politeness, accuracy. \"The\ndeveloper should have been more careful\" has no implementation.\n\n**This is an explicit tool.** Invoke it with a slash command or by asking for it by name.\nThat is the supported path and it works.\n\nIt does not auto-trigger. Ten realistic queries, among them workspace-deletion UX, tenant\nisolation, an agent ignoring CLAUDE.md and a Friday column-drop migration, were put to fresh\nagents with the plugin installed and no hint it existed. They are the ten `conversational`\n\ncases in\n[ benchmarks/trigger-cases.json](/rainmanjam/poka-yoke/blob/main/benchmarks/trigger-cases.json). None invoked a poka-yoke skill; one reasoned about\nskills explicitly and picked\n\n`hookify`\n\ninstead.**That is the platform, not these descriptions.** Skills are documented as model-invoked and\nfrequently are not: [anthropics/claude-code#9716](https://github.com/anthropics/claude-code/issues/9716)\ncollects reports of skills ignored even when the query exactly matches the description, and\n[Scott Spence's write-up](https://scottspence.com/posts/claude-code-skills-dont-auto-activate)\ndocuments the same thing independently. Running the skill-creator description optimizer here\nchanged nothing across three rewrites.\n\nHow the field has responded, and what it costs:\n\n| Approach | Example | Cost |\n|---|---|---|\n| A forceful meta-skill injected every session |\n\"even a 1% chance a skill might apply… you do not have a choice\" |\n\n`UserPromptSubmit`\n\nhook naming the specific skill[claude-code-infrastructure-showcase](https://github.com/diet103/claude-code-infrastructure-showcase)*gentle*reminder is provably ignoredA hook is shipped for the middle option, `assets/devices/claude-hooks/suggest_poka_yoke.py`\n\n.\nIt matches the prompt against each mode's vocabulary and injects an instruction naming that\nskill. Tested: one matching prompt per mode routes correctly, and a shared set of four\nunrelated prompts routes to nothing, in `tests/test_detector.py`\n\n. That is a near-miss for the\nrouter, not one per mode. It is\n**Warning rung, not Control**: the injected instruction is still an instruction, and our own\nconclusion after living with it is that for anything important you invoke explicitly anyway.\n\nWhich is the honest summary of the whole area: **for a method you reach for deliberately, the\nslash command is the device and everything else is a convenience.**\n\nOne thing those runs surfaced that is worth knowing before installing anything: **the no-skill\nbaseline is strong.** Unprompted, current models already reach for row-level security with\n`FORCE`\n\n, the pooled-connection trap, expand/contract migrations, soft-delete-with-undo over\nconfirmation dialogs, and hooks over prose. The [benchmarks](#benchmarks) measure what this\nadds on top: model baselines run 58.2% to 92.7%, rising to 71.1% to 97.0%. Real, but an improvement to something already\ncompetent rather than a missing capability.\n\nThirteen scenarios run against four Claude models under two configurations, **445 runs,\nblind-graded** against pre-written assertions. Two non-Claude runtimes add 146 more, reported\n[separately below](#does-it-work-outside-claude-code); every figure in this section is the\nClaude matrix alone. Nine of the thirteen scenarios are a message in which the user has **already applied or\nproposed a fix that is insufficient**, so agreeing with them scores badly. The other four,\n`design`\n\nand the three `build-*`\n\nprompts, are greenfield: nobody has raised a concern, and\nthey measure what the model reaches for unprompted. This measures pushback, not recall.\n\n| Model | Baseline | With skill | Delta | 95% CI on the delta | Time |\n|---|---|---|---|---|---|\nFable 5 |\n88.7% (sd 11.8) | 97.0% (sd 4.0) | +8.3 pp |\n[+1.1, +15.5] | 64s → 91s |\nOpus 5 |\n92.7% (sd 7.9) | 96.4% (sd 5.5) | +3.6 pp |\n[+0.2, +7.0] | 130s → 172s |\nSonnet 5 |\n79.8% (sd 12.7) | 88.5% (sd 6.1) | +8.6 pp |\n[+2.2, +15.1] | 88s → 129s |\nHaiku 4.5 |\n58.2% (sd 16.7) | 71.1% (sd 24.1) | +12.9 pp |\n[+0.9, +24.9] | 40s → 60s |\n\n`sd`\n\nis the standard deviation of pass rates **across scenarios**: how unevenly a model\nperforms over the suite. It is not a confidence interval. The CI column is: a 95% interval on\nthe paired per-scenario difference, which is the statistic that answers \"does this help\",\nbecause scenarios differ far more in difficulty than runs do in noise.\n\nAcross the 52 scenario×model cells: **30 improved, 13 unchanged, 9 regressed**, mean\n**+8.3 pp**. See [how this compares to other skill benchmarks](/rainmanjam/poka-yoke/blob/main/docs/benchmark-comparison.md).\n\n**All four intervals clear zero, and two of them barely.** Opus 5's lower bound is +0.2 pp and\nHaiku 4.5's is +0.9 pp. The effect is real and, on the frontier models, small.\n\n**Benefit tracks available headroom, measurably.** Across the 52 cells, a cell's baseline\ncorrelates with its gain at **r = −0.52**: cells starting below 50% gain +19 pp on average,\ncells starting above 95% lose 0.9 pp. Averaged over the suite the skills close **36% of the\nremaining headroom**. That is what a skill encoding a *method* looks like, as opposed to one\nsupplying missing knowledge. It cannot help a model that was already going to do the thing.\n\n**The largest movements are on the weakest model and the build scenarios.** `ops`\n\non\nHaiku 4.5 goes **29% → 92%**, `build-endpoint`\n\non Fable 5 **61% → 100%**, and on Sonnet 5\n`agent-guardrails`\n\n**59% → 91%** and `guardrails`\n\n**66% → 96%**: the scenarios about\nbuilding devices rather than finding hazards.\n\n**Consistency improves except where it is worst.** Fable 5's spread falls from **11.8 to\n4.0** and Sonnet 5's from **12.7 to 6.1**, while Haiku 4.5's *rises* from **16.7 to 24.1**.\nThe skill makes Haiku better on average and less predictable, which is a real cost.\n\nNine of 52 cells came out negative. Under the null of no effect at all, roughly **18** would:\nsimulating from the real per-cell run counts and the median of 8 assertions per run puts the\n95% range at 12 to 25. Nine is below that range, so the count of regressions is evidence the\neffect is consistently positive, not evidence of hidden harm.\n\nThese four fell by more than 5 points. They are listed because they are where to look, not because any one of them is callable on its own:\n\n| Cell | Baseline → skill | runs |\n|---|---|---|\n`build-agent-feature` on Haiku 4.5 |\n62% → 31% |\nn=2 |\n`build-endpoint` on Haiku 4.5 |\n44% → 33% |\nn=2 |\n`audit` on Fable 5 |\n100% → 94% | n=3 |\n`authz` on Sonnet 5 |\n94% → 89% | n=7 |\n\n**No individual cell here supports a claim.** At these sizes, calling a 30-point effect real\nneeds about 32 runs and a 10-point effect about 199. The two Haiku `build-*`\n\nresults are worth\nwatching because they point the same way as a mechanism that would make sense, a small model\nspending its output on the method rather than the thing, but two runs cannot establish it. That\nis a hypothesis for the next sweep, not a finding.\n\n`authz`\n\non Sonnet 5 was first reported as a 16-point regression. Investigating it found two defects, both\nin the measuring apparatus rather than the skill.\n\n**The aggregate was reading half the data.** `aggregate()`\n\nlooped `range(1, --runs + 1)`\n\nand\n`--runs`\n\ndefaults to 3, so re-aggregating after a seven-run sweep counted 264 of 486 runs with\nno warning. It now reads the run directories that exist.\n\n**One assertion tested layout, not detection.** *\"Notes the SQL injection separately from the\nscoping issue\"* was failing responses that identified the injection in a heading, because they\npresented it as a compounding factor within the tenant-scoping finding. It now asks for the\ninjection to be identified as a hazard distinct in kind, anywhere in the response.\n\nThree tests were added so neither recurs, and a third covers a bug introduced while fixing\nthem: a regrade that deleted the old grading first destroyed 11 of them when the grader call\nfailed, and because a missing grading merely shrinks a cell, the summary printed a model short\nwithout complaint. `tests/test_portability.py`\n\nnow fails if a cell's `n`\n\ndisagrees with the\nruns on disk, if any grading was scored against a superseded checklist, or if any stored\nresponse has no grading at all.\n\n`agent-guardrails`\n\nneeded one thing more. Haiku's failing runs opened *\"Nothing. You're doing\nnothing wrong\"*, answering the rhetorical question, delivering the skill's thesis, and\nstopping. The skill's central insight was quotable enough to crowd out the remedy. Adding\n\"the diagnosis is not the answer, state it in a sentence, then spend the rest on the\nreplacement\" fixed it.\n\n| at the time of the fix | in the committed aggregate | |\n|---|---|---|\n`ops` / Haiku 4.5 |\n58% | 92% |\n`ops` / Sonnet 5 |\n71% | 88% |\n`agent-guardrails` / Sonnet 5 |\n54% | 91% |\n`agent-guardrails` / Opus 5 |\n71% | 100% |\n`agent-guardrails` / Haiku 4.5 |\n38% | 33% |\n\n**The agent-guardrails fix did not hold on Haiku 4.5.** It was measured at 88% when the\nrestructuring landed; the runs committed here score 4/8, 4/8 and 0/8, which is 33% and worse\nthan the 38% it started from. The other four cells held or improved. The earlier figure is\nleft in the left-hand column rather than deleted, because a fix that stopped working is worth\nmore to a reader than a table that only shows the times it did.\n\n** llm on Sonnet 5 is flat at 90% → 89%** across seven runs each: the earlier four-point\nregression there was measurement noise at n=3, not an effect. The live regressions are the\nfour listed above.\n\nThe skill makes the model read the router, the matching sub-skill, and often a reference file before answering, then produce a fuller answer. Both show up as cost.\n\n| Task shape | Model | Δ pass rate | Output length | Wall-clock |\n|---|---|---|---|---|\n| Advice | Fable 5 | +6.3 pp | 1.26× | 1.44× |\n| Advice | Opus 5 | +3.0 pp | 1.22× | 1.36× |\n| Advice | Sonnet 5 | +8.4 pp | 1.24× | 1.16× |\n| Advice | Haiku 4.5 | +19.3 pp | 1.84× | 1.47× |\nBuild |\nFable 5 |\n+14.8 pp |\n1.31× | 1.45× |\nBuild |\nOpus 5 | +5.6 pp | 1.14× | 1.34× |\nBuild |\nSonnet 5 | +9.5 pp | 1.30× | 1.90× |\nBuild |\nHaiku 4.5 |\n−8.6 pp |\n1.14× | 2.12× |\n\nTwo things to read off this. **The build tasks split the fleet.** Fable 5 gains most there\n(+14.8 pp) while Haiku 4.5 is the only cell in the whole suite that is clearly negative\n(−8.6 pp) at more than double the wall-clock, asked to build something, the smallest model\nspends its budget on the method and ships less of the thing. And **the delta is not bought\nwith extra output**: across all 52 scenario×model cells, the correlation between how much\nlonger the answer got and how much better it scored is only **r = 0.30**. Length is not the\nmechanism.\n\nWhat *does* predict the gain is how much the baseline was missing:\n\nBaseline vs. delta: r = −0.52across 52 cells. Headroom predicts gain.\n\nHeadroom explains most of the variation, but not all of it, and the exception matters. On the\nrefund-endpoint task Opus 5 goes **83.3% → 100%** (+16.7 pp, closing all of its headroom) while\nHaiku 4.5 goes **44.4% → 33.3%**: the model with the *most* headroom is the one that gets\nworse. Headroom sets the ceiling on what a method can add; it does not guarantee the model can\nuse the method and still deliver the artifact.\n\nThat is the boundary of the claim. On advice-shaped tasks the skills help every model, most where the baseline is weakest. On build-shaped tasks they help three models and hurt the smallest one, because reading and applying a method competes with writing the code.\n\n**The practical rule:** the cost is roughly constant and the benefit is not, so this pays for\nitself in proportion to the gap between the model doing the work and what the task demands.\nOn a frontier model writing a well-trodden endpoint it closes the remaining gap outright\n(83.3% → 100.0%). On the fastest model doing the same work it currently costs more than it\nreturns (44.4% → 33.3%), which is the one configuration this plugin does not recommend.\n\n*Measured as output words and wall-clock, not token counts: the harness does not read token\nusage back from the CLI, so treat these as proportional rather than exact.*\n\nThe plugin ships to 19 runtimes and only Claude Code was ever measured. Two more now are, OpenAI Codex (`gpt-5.6-terra`\n\n, read-only sandbox) and agy (Gemini 3.1 Pro, plan mode), run\nthrough the same harness, the same scenarios, and the same blind grader.\n\n| Runtime | Baseline | With skill | Delta | Scenarios |\n|---|---|---|---|---|\nCodex |\n74.2% (sd 17.7) | 91.0% (sd 9.8) | +16.8 pp |\n13 of 13 |\nagy |\n64.6% (sd 13.9) | 78.1% (sd 13.2) | +13.5 pp |\n12 of 13 |\n\n**These are the two largest gains in the suite, and they fit the same curve.** Codex and agy\nstart at 74.2% and 64.6%: the lowest baselines after Haiku 4.5, and gain the most. The\neffect tracking headroom is not a Claude artefact; it holds across three vendors.\n\nThey are reported separately rather than folded into the headline mean, for two reasons. agy's\ncolumn is short one scenario, for the reason stated in `benchmarks/results/benchmark.md`\n\n. And the\nruntimes answer at very different lengths: a Codex reply runs 50–200 words where a Claude\nanswer runs 400–800, so the same checklist lands differently against each, and only the\ndelta *within* a runtime is a fair comparison.\n\nOne finding worth more than the numbers: ** audit cannot run on agy at all.** That skill\ninvokes the bundled detector, and agy's plan mode refuses to execute a command. Any runtime\nthat blocks execution can use the other ten skills and not that one.\n\nGrading is by a model (Haiku 4.5), not a human. Assertions were written by the skill's author, before the runs, not fitted to them, but that is not independence. Cells hold 1–7 runs, so the\nper-scenario means behind the intervals are themselves noisy, and **three cells hold a single\nrun**: `build-agent-feature`\n\nfor Fable 5 baseline and both Opus 5 configurations, where repeated\nattempts across three separate windows were lost to API rate limits. A one-run cell has no\nvariance estimate and still carries full weight in its model's mean. All three sit at 100%,\nso a second run could only confirm or lower them. Claude Code, Codex and Antigravity were measured;\nthe [other install targets](/rainmanjam/poka-yoke/blob/main/docs/install.md) are untested.\n\nThis supersedes earlier benchmark rounds.One reportedno gain on Opusfrom saturated assertions and a grader that knew which configuration it was scoring; both flaws are fixed. A later round under-reported Sonnet 5 and invented an`authz`\n\nregression because the aggregate was reading three runs per cell out of seven. Where this README and an older number disagree, this one is computed from the 591 runs currently in`benchmarks/results/`\n\n.\n\nEvery run, grading, and timing is in [ benchmarks/results/](/rainmanjam/poka-yoke/blob/main/benchmarks/results), and the\nharness is\n\n[, re-run it and check the numbers.](/rainmanjam/poka-yoke/blob/main/benchmarks/run.py)\n\n`benchmarks/run.py`\n\n```\n.claude-plugin/     marketplace catalog\nplugins/poka-yoke/  the plugin: skills, references, scripts, device templates\n                    plus a manifest per runtime (.codex-plugin/, .cursor-plugin/, …)\nscripts/            repo tooling; generates every platform manifest from one source\ndocs/               method, install guide, assets\nbenchmarks/         benchmark prompts, assertions, and fixtures\ntests/              the devices that guard the devices\n.github/workflows/  validate.yml guards every PR; release.yml turns a tag into a release\nAGENTS.md           symlink to CLAUDE.md; the other context files are one-line pointers\n```\n\n[ plugins/poka-yoke/README.md](/rainmanjam/poka-yoke/blob/main/plugins/poka-yoke/README.md) has the full tree and notes on\nediting the skills.\n\n- Shigeo Shingo,\n*Zero Quality Control: Source Inspection and the Poka-Yoke System*(1986): the origin [Make illegal states unrepresentable](https://deviq.com/principles/make-illegal-states-unrepresentable/)·[the counter-argument](https://www.seangoedecke.com/invalid-states/), which is worth taking seriously- Don Norman,\n*The Design of Everyday Things*, forcing functions [Applying mistake-proofing to software](https://mistakeproofing.com/applying-mistake-proofing-to-software/)[codehackerr/poka-yoke](https://github.com/codehackerr/poka-yoke)·[bryanhunter/poka-yoke](https://github.com/bryanhunter/poka-yoke)\n\nNew hazards and devices are welcome, see [CONTRIBUTING.md](/rainmanjam/poka-yoke/blob/main/CONTRIBUTING.md) and\n[CODE_OF_CONDUCT.md](/rainmanjam/poka-yoke/blob/main/CODE_OF_CONDUCT.md). The bar for a new catalog entry: a specific wrong\naction a person can take, a consequence, and a device with an honest rung.\n\nIf you run poka-yoke on a runtime other than Claude Code, a session transcript in an issue is the most useful thing you can send. Those manifests are verified structurally in CI but not behaviourally by us.\n\nReleases are cut by pushing a tag; `.github/workflows/release.yml`\n\ndoes the rest, and\n[RELEASING.md](/rainmanjam/poka-yoke/blob/main/RELEASING.md) explains what it refuses and why.\n\nMIT, see [LICENSE](/rainmanjam/poka-yoke/blob/main/LICENSE).", "url": "https://wpnews.pro/news/poka-yoke-mistake-proofing-claude-code-skill-for-software", "canonical_source": "https://github.com/rainmanjam/poka-yoke", "published_at": "2026-08-25 00:14:15+00:00", "updated_at": "2026-08-25 00:42:44.871687+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "ai-research"], "entities": ["rainmanjam", "Claude Code", "Fable 5", "Opus 5", "Sonnet 5", "Haiku 4.5", "Shigeo Shingo"], "alternates": {"html": "https://wpnews.pro/news/poka-yoke-mistake-proofing-claude-code-skill-for-software", "markdown": "https://wpnews.pro/news/poka-yoke-mistake-proofing-claude-code-skill-for-software.md", "text": "https://wpnews.pro/news/poka-yoke-mistake-proofing-claude-code-skill-for-software.txt", "jsonld": "https://wpnews.pro/news/poka-yoke-mistake-proofing-claude-code-skill-for-software.jsonld"}}