{"slug": "forespec-catches-what-your-ai-coder-didn-t-know-to-ask", "title": "Forespec – catches what your AI coder didn't know to ask", "summary": "Forespec, an early-build tool from developer SteveWeed79, runs inside AI coding agents such as Claude Code to surface non-obvious requirements — atomic stock holds, tenant isolation, payment webhook authenticity — before code is written, then grades what was built. The tool ships five archetypes, a reasoning verifier, a plan/interrogator, a PR gate, and a greenfield on-ramp, and can be tried with no install or API key via the command \"npx forespec demo\"; standalone grading requires an API key, while the Claude Code plugin path runs on an existing Claude Code subscription.", "body_md": "*Forespec keeps you pointed in the right direction — **at the start** of a feature, **along\nthe way** as it grows, and **over time** by remembering your past results to catch code\ndegradation before it compounds. It's the domain **foresight** a senior engineer brings,\ncodified, and kept live through your whole build.*\n\nYou ship fast with AI coding tools. They build what you *ask* — not the non-obvious thing your\n*kind* of app actually requires: an ecommerce checkout needs an **atomic stock hold**; a SaaS\nneeds **tenant isolation**; a payment webhook needs **authenticity**. Miss one and you find out\nin month three, doing surgery on a live flow. Forespec surfaces those requirements **before you\nbuild**, hands your AI coder a gotcha-aware spec, then grades what got built and tracks how each\npart moves run-over-run — so the foresight arrives on time, and stays live.\n\nForespec is **not** a security scanner. Security is one row of what it checks; the rest is\ncorrectness, data-modeling, reliability, and design — the whole backbone your archetype\nrequires. It runs **inside the coding agent you already use** (Claude Code first), so the\nforesight lands while the code is being written rather than in a review three days later.\n\n**Status: early build.** Five archetypes, the reasoning verifier, the plan/interrogator, the\nPR gate, and the greenfield on-ramp are here and runnable. New to this kind of project?\n[`SETUP.md`](https://github.com/SteveWeed79/forespec/blob/main/SETUP.md) gets you going on Windows, step by step.\n\nNo install, no API key, nothing to configure — this replays the verifier on a bundled vulnerable-checkout example so you can see exactly what a real grade looks like:\n\n```\nnpx forespec demo\n```\n\nIt flags the non-obvious criticals (a Stripe call with **no idempotency key** → double\ncharges; a **stock race**; an order marked *paid* on the unverified client return), **passes**\nthe one that's actually fine (card data never touches your server), and surfaces a required\npiece you *haven't built yet* — the discernment a grader you'd trust with \"is this shippable?\"\nhas to earn. Then point it at your own repo:\n\nThe shortest path to a real grade on your own code. No API key, no metered cost — it runs on the Claude Code subscription you already have:\n\n```\n/plugin marketplace add SteveWeed79/forespec\n/plugin install forespec@forespec\n/forespec:plan add checkout     # before you build — what does this actually require?\n/forespec:verify                # after you build — what's shippable, what's not\n```\n\nIt also loads on its own the moment you start writing payment, auth, tenancy, upload, LLM, or\nSupabase/Firebase code, so the requirement arrives at *write* time — an idempotency key passed\non the payment intent is an argument; retrofitted into a live payment path it's surgery. Full\ndetails, including how to drive it from any other agent:\n[`docs/claude-code-plugin.md`](https://github.com/SteveWeed79/forespec/blob/main/docs/claude-code-plugin.md).\n\n**This is also the better grader**, not a cheap fallback. The API path packs keyword-ranked\nfiles into a character budget and grades the blob, so it can only cite a file path. An agent\nhas grep and read: it follows the call chain into the middleware that supposedly verifies the\nsignature, and cites `file:line` — a finding you can fix instead of one you have to go\nre-investigate.\n\nFor scripting, or when there's no agent in the loop:\n\n```\n# New/empty repo — DECLARE what you're building; Forespec points you and writes a build plan:\nforespec start \"an online store with checkout\"\n\n# Existing repo — detect the archetype from your code's metadata instead:\nforespec init\n\n# Before a feature — interrogate what it actually requires (foresight before you build):\nforespec plan \"add checkout flow\"\n\n# After you build — grade the backbone, and see how each checkpoint moved vs last time:\nforespec verify              # standalone grading needs an API key — see below\nforespec verify --html       # …drop a visual report you can open in a browser\nforespec gate --help         # wire the PR/CI gate that comments on every pull request\n```\n\n`start` and `plan` are the **foresight-before-build** half: they surface the non-obvious,\narchetype-required properties — the *\"decide first\"* questions and acceptance criteria — and\nhand them to your AI coder as an ordered, dangerous-pieces-first spec. `verify` and `gate` are\nthe **keeps-it-honest** half: they grade those same checkpoints on the real code, flag what's\nunsafe, surface the required backbone you *haven't built yet*, and — reading the calibration\nstore — show how each checkpoint moved **since your last run**, so a regression doesn't slip\nthrough. Over time `forespec proficiency` reads how much judgment you've shown per domain\n(self-facing only) and `plan` adapts how much it explains — full where you're learning, terse\nwhere you're fluent.\n\n`start`/` init` read only metadata (dependencies, paths, schema-model names) or your one-line\ndescription — never your code — to pick the archetype.\n\n**Grading needs a reasoning verifier**, and there are two ways to get one:\n\n- **The plugin** (above) — your coding agent grades the repo and hands the verdicts back\nthrough the same roll-up, gaps report and calibration store. No key, no cost, and it can\ncite`file:line` . Run it as`/forespec:verify` . Measured on the same corpus as the API\npath:**0 false-greens across 152 critical-bad trials → 95% upper bound ≤2.0%** , with\n100% outcome agreement across two independent runs. Caveats — that measures the grading\ncontract on snippets, not the repo-navigation advantage, and your session's model is the\ngrader — are in[`VALIDATION-NOTES.md`](https://github.com/SteveWeed79/forespec/blob/main/VALIDATION-NOTES.md) .\n- **An API key** — set`ANTHROPIC_API_KEY` +`ANTHROPIC_MODEL` and`verify` calls the model\ndirectly. It carries the measured bar: 0 false-greens on 52 critical bad cases, rule-of-three 95% upper bound ≤ 2.9%\n(see[`VALIDATION-NOTES.md`](https://github.com/SteveWeed79/forespec/blob/main/VALIDATION-NOTES.md) ). That bar covers the ecommerce/universal\ncorpus; the newer`saas` /`ai-app` /`baas` archetypes are**first-pass** validated (full\nrule-of-three pending).\n\n**With neither, `verify` refuses.** It won't fake a grade — a page of keyword-matched \"risky\"\nverdicts looks like a verdict no matter how it's labelled, and the point of this tool is a\nnumber you can act on. The `mock` keyword baseline still exists as the dumb bar a real verifier\nhas to beat, but you have to ask for it by name (`--adapter mock`), and it can never certify a\nmerge. Full walkthrough: [`repo-verify/README.md`](https://github.com/SteveWeed79/forespec/blob/main/repo-verify/README.md).\n\n**CI runs on your subscription too.** `claude setup-token` produces an OAuth token that\nauthenticates with a Pro/Max/Team/Enterprise plan, so the PR gate bills against the plan you\nalready have rather than a metered key —\n[`docs/ci-gate-agent.md`](https://github.com/SteveWeed79/forespec/blob/main/docs/ci-gate-agent.md). The merge decision stays deterministic:\nthe agent produces verdicts, `forespec gate` reads them and decides.\n\n**What works with no verifier at all**, right now, no setup: `forespec demo` (a real graded run\non a bundled example), `forespec plan \"<feature>\"` (what the feature actually requires, before\nyou build it), `forespec init` (archetype detection from metadata), and `forespec checkpoints`\n(the standard itself, as JSON). The foresight half of the product costs nothing to try.\n\n- **Point** —`start` /`plan` interrogate the domain and emit a gotcha-aware spec (atomic hold\nbefore Stripe, data-model shape before the features built on it), most-foundational first.\n- **Build** — you, or your AI coder, build against that spec.\n- **Verify** —`verify` /`gate` grade what actually got built, flag what's unsafe, and name the\nrequired backbone you haven't reached yet.\n- **Remember** — every run writes to a local calibration store behind a strict**pattern /\ninstance wall** , so the next run can tell you what*moved* (catching a regression) and, over\ntime, sharpen the foresight itself.\n\nThat last step is the point: the standard isn't a static checklist — it **compounds** on your\nwork (and, opt-in later, across a shared pattern pool), while your project's specifics never\nleave your machine.\n\nPointed at 8 public OSS repositories it had never seen — spanning all five archetypes, including a\n4,332-file Python codebase — the plugin's grader produced **147 verdicts: 18 findings, 112 passes,\n17 N/A**. Every finding was then checked against the source by hand:\n\n- **12 of 18 findings hand-verified. 0 fabrications.** Every`file:line` pointed at real code that\nsaid what the verdict claimed.\n- **0 of 147 verdicts came back without evidence.** Every one cited`file:line` .\n- **It comes back clean on clean code.** Documenso, LibreChat and supabase-js produced zero\nfindings; four of Documenso's strongest passes were falsification-tested and held.\n- The full ledger — including **four defects it found in Forespec itself** — is in[`docs/oss-audit-2026-09.md`](https://github.com/SteveWeed79/forespec/blob/main/docs/oss-audit-2026-09.md) . Reproduce any run with`node verifier-eval/repo-audit.mjs --repo <path>` .\n\nPointed at a **real production ecommerce app** (a codebase it had never seen, not a fixture),\n`forespec verify` returned one blocking critical: the Stripe **checkout-session creation call\ncarried no idempotency key**, so a double-click or a client retry could open two charges for one\ncart. Confirmed real by direct code review. It also caught a subtler one *while passing that\ncheckpoint at level 9* — a refund idempotency key with no per-refund nonce, so two equal-amount\npartial refunds collide and reconcile wrong.\n\nEvery finding was then independently re-checked against the actual source. The result, recorded\nhonestly in [`docs/real-repo-audit-2026-07.md`](https://github.com/SteveWeed79/forespec/blob/main/docs/real-repo-audit-2026-07.md):\n\n- **0 fabricated findings, 0 false-greens** — every flag pointed at real code.\n- The bias is **over-severity, not fabrication** — it would rather flag something you've already\nhandled than miss something you haven't. That's the safe direction, and the candor a grader you\ntrust with*\"is this shippable?\"* has to earn.\n\nNot \"perfect\" — honest. That's the whole point.\n\n| File | What it is | \n|---|---|\n| [`docs/ci-gate-agent.md`](https://github.com/SteveWeed79/forespec/blob/main/docs/ci-gate-agent.md) | The PR gate on a Claude subscription instead of an API key — and why the merge decision stays with `pr-gate.mjs` , not the model. | \n| [`docs/oss-audit-2026-09.md`](https://github.com/SteveWeed79/forespec/blob/main/docs/oss-audit-2026-09.md) | **The field report.** 8 public OSS repos graded by the plugin, every finding checked by hand — including the four defects it found in Forespec itself. | \n| [`docs/claude-code-plugin.md`](https://github.com/SteveWeed79/forespec/blob/main/docs/claude-code-plugin.md) | **The front door.** How the plugin turns your coding agent into the verifier — no API key — and how to drive the same path from any other agent. | \n| [`FORESPEC-2.md`](https://github.com/SteveWeed79/forespec/blob/main/FORESPEC-2.md) | The vision: the full architecture and the moat argument. **Superseded on build *sequence*** by the build order below. | \n| [`forespec.buildorder-2.md`](https://github.com/SteveWeed79/forespec/blob/main/forespec.buildorder-2.md) | **The authoritative roadmap.** Phases 0–7, verifier-first, each phase shippable on its own. When any doc disagrees on*what to build in what order* , this one governs. | \n| [`forespec.calibration-1.md`](https://github.com/SteveWeed79/forespec/blob/main/forespec.calibration-1.md) | The calibration loop that turns invented weights into ones earned on real work, and the seam that lets solo data later join a shared pool without a rewrite. | \n| [`library/`](https://github.com/SteveWeed79/forespec/blob/main/library) | The **shared checkpoint library** — every checkpoint definition (auth, payment, data, design, ai, baas, …), authored once and reused across archetypes.`resolve.mjs` composes a manifest + the library into a full archetype. | \n| [`archetype.ecommerce.json`](https://github.com/SteveWeed79/forespec/blob/main/archetype.ecommerce.json) | The ecommerce **archetype manifest** : 20 backbone + 7 design checkpoints from the library, each with its severity for this domain. Resolves to the durable standard a verifier grades against. | \n| [`archetype.ecommerce.design.json`](https://github.com/SteveWeed79/forespec/blob/main/archetype.ecommerce.design.json) | The **instrumented** design layer: design checkpoints decomposed into weighted, measurable sub-signals → a computed 0–10 composite. Its`model_scored` signals are deferred experiments until calibration earns them. | \n| [`archetype.saas.json`](https://github.com/SteveWeed79/forespec/blob/main/archetype.saas.json) | The SaaS / subscription manifest — 26 checkpoints, all but 3 **reused** , 3 SaaS-specific (tenant isolation, entitlement integrity, subscription lifecycle). | \n| [`archetype.portfolio.json`](https://github.com/SteveWeed79/forespec/blob/main/archetype.portfolio.json) | The portfolio / content manifest — 14 checkpoints, **100% composed** from the shared library (design + web + the universal set), zero new authoring. | \n| [`archetype.ai-app.json`](https://github.com/SteveWeed79/forespec/blob/main/archetype.ai-app.json) | The **AI / LLM app** manifest — 12 checkpoints,**5 AI-specific** (prompt injection, output handling, tool-use safety, cost controls, data boundary) + 7 reused. | \n| [`archetype.baas.json`](https://github.com/SteveWeed79/forespec/blob/main/archetype.baas.json) | The **Backend-as-a-Service (Supabase / Firebase)** manifest — 10 checkpoints,**3 BaaS-specific** (RLS enforced, client trust boundary, privileged-key exposure) + 7 reused. | \n\nReading order: `FORESPEC-2.md` (the why) → `forespec.buildorder-2.md` (the how and the order,\nthe plan of record) → `forespec.calibration-1.md` (the layer that keeps every score honest over\ntime) → [`library/`](https://github.com/SteveWeed79/forespec/blob/main/library) + `archetype.ecommerce.json` (the standard itself).\n\n- **Pattern / instance wall.** Transferable patterns and never-leaves-the-project instance data\nlive in separate stores from the first write — the legal and ethical line, and the thing that\nmakes a shared pattern pool safe to opt into later.\n- **Honesty mechanic.** Every score reports its level, the gap to the next, and its basis. A\nscore that can't state its basis doesn't ship.\n- **Stable, namespaced checkpoint ids** (e.g.`payment.webhook_authenticity` ,`ecommerce.checkout.atomic_stock_hold` ) are permanent contracts. Archetypes are*versioned* ;\ncheckpoints are*never silently renamed* — that's how calibration history survives.\n- **Library + manifest composition.** Checkpoints are defined once in[`library/`](https://github.com/SteveWeed79/forespec/blob/main/library) and*composed* per archetype (a manifest of`{ ref, severity }` ), so a fix to a shared\ncheckpoint improves every archetype and a new archetype reuses instead of copies.\n\n- **Doc naming.** Prose specs are`forespec.<topic>-<n>.md` ;`FORESPEC-2.md` is the top-level\nvision. The trailing`-1` /`-2` are iteration numbers (higher = later).\n- **Archetype versioning.** Each manifest and library file carries a semver`version` / format\ntag. Checkpoint*ids* are permanent contracts — bump the version when a definition changes,\nnever rename an id.\n- **`$schema`.** Files declare a` forespec/…` format tag (`forespec/archetype/v2` ,`forespec/checkpoint-library/v1` , …) — the engine's internal contract, stable across the\nbrand. JSON Schemas that validate them live in[`schemas/`](https://github.com/SteveWeed79/forespec/blob/main/schemas) .\n\n- [`bin/forespec.mjs`](https://github.com/SteveWeed79/forespec/blob/main/bin) — the unified` forespec` CLI (`start` /`init` /`detect` /`plan` /`verify` — add`--html` for a visual report — /`design` /`gate` /`feedback` /`calibrate` /`proficiency` ), exposed for`npx` .\n- [`library/`](https://github.com/SteveWeed79/forespec/blob/main/library) — shared checkpoint library +`resolve.mjs` (compose a manifest +\nlibrary into a full archetype:`node library/resolve.mjs archetype.ecommerce.json` ).\n- [`schemas/`](https://github.com/SteveWeed79/forespec/blob/main/schemas) — JSON Schema for the library, the manifest, the resolved archetype,\nand the instrumented design layer, plus a zero-dependency invariant validator\n(`node schemas/validate.mjs` ).\n- [`verifier-eval/`](https://github.com/SteveWeed79/forespec/blob/main/verifier-eval) — the verifier-accuracy harness. A labeled good/bad fixture\ncorpus for the critical backbone checkpoints, plus a runner that measures a verifier's\nprecision/recall and**false-green rate** (`node verifier-eval/run-eval.mjs` ). This is how \"is\nthe verifier trustworthy?\" becomes a number instead of a hope.\n- [`repo-verify/`](https://github.com/SteveWeed79/forespec/blob/main/repo-verify) — the product surface: point the verifier at a*whole real\nrepo* . Archetype detection + the greenfield`start` on-ramp, the verifier CLI, the calibration\nstore (the pattern/instance wall, made physical), and the git-aware**PR gate** + drop-in\nGitHub Action (`action.yml` ). Start at[`repo-verify/README.md`](https://github.com/SteveWeed79/forespec/blob/main/repo-verify/README.md) .\n\n**Business Source License 1.1** — see [`LICENSE`](https://github.com/SteveWeed79/forespec/blob/main/LICENSE). Free to use and self-host for any\npurpose, including commercially; the one reserved right is offering **Forespec itself as a\ncompeting hosted service**. Each released version converts to **Apache 2.0** four years after it\nships. The commitment it encodes: the local core is **free and fully useful forever** — never\ncrippled to force an upgrade, no dark patterns — while paid, hosted plans fund the ongoing work\nthat keeps the standard trustworthy; priced fairly, no lock-in.", "url": "https://wpnews.pro/news/forespec-catches-what-your-ai-coder-didn-t-know-to-ask", "canonical_source": "https://github.com/SteveWeed79/forespec", "published_at": "2026-09-16 17:28:04+00:00", "updated_at": "2026-09-16 17:44:05.602816+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools", "ai-products"], "entities": ["Forespec", "SteveWeed79", "Claude Code", "Stripe", "Supabase", "Firebase"], "alternates": {"html": "https://wpnews.pro/news/forespec-catches-what-your-ai-coder-didn-t-know-to-ask", "markdown": "https://wpnews.pro/news/forespec-catches-what-your-ai-coder-didn-t-know-to-ask.md", "text": "https://wpnews.pro/news/forespec-catches-what-your-ai-coder-didn-t-know-to-ask.txt", "jsonld": "https://wpnews.pro/news/forespec-catches-what-your-ai-coder-didn-t-know-to-ask.jsonld"}}