{"slug": "executable-contracts-as-guardrails-for-ai-generated-code", "title": "Executable Contracts as Guardrails for AI-Generated Code", "summary": "A developer integrated Specmatic's executable contracts into their natural-language-to-SQL engine DATABASE-MANAGER to guardrail AI-generated code. The contracts automatically detect API contract drift caused by AI coding agents, catching silent changes like field renames before they break integrations. The approach extends the project's human-in-the-loop safety model from database writes to API contracts.", "body_md": "*Submitted for the Specmatic Full Stack AI Engineering Intern challenge.*\n\nI build with AI coding agents every day — Claude Code, Cursor. They're incredible at velocity. But they have one quiet, expensive failure mode: **they drift.** You ask for a small change to an endpoint, and somewhere in the diff the agent renames a field, flips a status code, or drops a required parameter. The code still runs. The tests (if you have them) still pass. And then three layers downstream — in the frontend, in a consumer service — something breaks, and you spend an afternoon debugging an integration bug that was never a logic bug at all. It was a **contract** bug.\n\nThat's the exact problem Specmatic's challenge asked me to explore: *can Spec-Driven Development and executable contracts improve AI-assisted software development?* After integrating Specmatic into one of my projects, my answer is an emphatic yes — and the reason is that **an executable contract is a guardrail for AI-generated code.**\n\n[DATABASE-MANAGER](https://github.com/AYON-ARYAN/DATABASE-MANAGER) is a natural-language-to-SQL engine I built that works across 8 database engines (SQLite, MySQL, PostgreSQL, MSSQL, Oracle, MongoDB, Cassandra, Redis). It has a Flask `/api`\n\nblueprint consumed by a React frontend, and — importantly — a **human-in-the-loop write-safety model**: any write/schema command doesn't execute directly. The API returns `{ \"needs_review\": true, ... }`\n\n, a human approves, and only then does `/api/execute`\n\nrun it, with snapshot rollback via `/api/undo`\n\n.\n\nSo the project already had a philosophy: *don't let the model run unchecked.* Specmatic let me extend that exact philosophy from **database writes** to the **API contract itself.**\n\nI wrote an OpenAPI contract (`api_contract.yaml`\n\n) describing the real API. The interesting part is the `/api/command`\n\nresponse, which is a *union* of three shapes — a READ result, a write staged for review, or an error:\n\n```\n/api/command:\n  post:\n    responses:\n      \"200\":\n        content:\n          application/json:\n            schema:\n              oneOf:\n                - $ref: \"#/components/schemas/ReadResult\"\n                - $ref: \"#/components/schemas/NeedsReview\"   # { needs_review: true, sql, explanation, task }\n                - $ref: \"#/components/schemas/Error\"\n```\n\nThis contract is now the single thing both the backend and the frontend agree on.\n\nWith Specmatic, the contract isn't documentation that rots — it's *executable*. Specmatic auto-generates positive and negative requests from the spec and verifies the live API conforms:\n\n```\ndocker run --rm --network host -v \"$PWD:/specs\" -w /specs \\\n  specmatic/specmatic:latest test --host localhost --port 5001\n```\n\nNo hand-written test cases. The spec *is* the test suite.\n\nSpecmatic can also *be* the API — a spec-conformant stub — so my React frontend can develop with no backend, no database, and no LLM keys running:\n\n```\ndocker run --rm --network host -v \"$PWD:/specs\" -w /specs \\\n  specmatic/specmatic:latest stub --port 9000\n```\n\nSame contract, two uses: it tests the provider *and* mocks it for the consumer. That's contract-driven development.\n\nHere's the demo that made it click for me. I asked an AI agent to \"add a confidence score and rename `needs_review`\n\nto `requiresReview`\n\n.\" It did — silently changing the response shape. To a human reviewer skimming the diff, it looks harmless. To the React frontend, it's a broken integration.\n\nThen I re-ran the contract test:\n\n**Specmatic caught it instantly** and told me *exactly* what drifted. A bug that would have surfaced as a confusing frontend error became a precise, one-line test failure — before it ever left my machine.\n\nBuilding with LLMs taught me a mantra: *it's 20% prompting and 80% guardrails.* The interesting engineering is everything you put **around** the model to make it safe. Specmatic is that idea applied to AI-assisted development itself:\n\nFor a tool like DATABASE-MANAGER — where I already gate database writes behind human review — adding a contract guardrail at the API layer felt like the natural second half of the same idea. Two guardrails, one principle: don't let generated actions run unchecked.\n\nThat's how Spec-Driven Development improves AI-assisted software engineering. It turns the AI's biggest weakness — confident, silent drift — into the cheapest possible signal: a failing test.\n\nAfter the first pass I pushed three Specmatic techniques further:\n\n**1. Inline + external examples.** Instead of letting Specmatic only check *shapes*, I added examples that pin *values* — so the contract test now verifies real cases: `admin1`\n\n→`ADMIN`\n\n, bad creds→`Invalid credentials`\n\n, and `editor1`\n\n→`EDITOR`\n\n(the last loaded as an external example file). One generic test became three value-checked ones, and the stub now returns lifelike data.\n\n**2. Schema-resiliency testing found a real bug.** Turning on generative negative tests, Specmatic mutated the login request — empty body, `username`\n\nas null / number / boolean — and expected a `4xx`\n\n. My API returned `200`\n\nfor all of them: it had **no input validation**, silently treating a malformed body as a failed login. 14 negative tests failed. I added body validation (`400`\n\non a non-object body or non-string fields), documented the `400`\n\n, and re-ran: **24/24 green**. The resiliency suite turned an invisible robustness gap into a fixed, tested guarantee.\n\n**3. Virtualizing the LLM itself.** Meridian's NL-to-SQL calls an LLM (Groq, which is OpenAI-compatible). Real tests would burn tokens on every CI run and tolerate non-determinism. So I wrote a contract for the LLM's `/v1/chat/completions`\n\n, ran `specmatic stub`\n\n, and pointed the app at it with a single env var.\n\n*How it's wired:* `core/llm_manager.py`\n\nreads `GROQ_API_URL`\n\n(unset in prod → real Groq); in CI a step boots `specmatic stub llm_contract.yaml`\n\n, sets `GROQ_API_URL`\n\nto it, and runs the real NL-to-SQL path against the stub — so the AI path is **offline, deterministic, and zero-token**. It's a *separate* CI step, after the API contract tests, because it virtualizes an upstream *dependency* the app **consumes**, not the app's own API. The stub spec is a deliberately reduced version of the real provider API — every deviation (and why) is documented in `LLM_CONTRACT_NOTES.md`\n\n. The same tool that guards *my* API also virtualizes the *dependency* my API consumes — exactly the kind of guardrail an AI-native codebase needs.\n\nAll three run in CI on every push, and the contract test reports **100% coverage** (generative negative tests cover the `400`\n\nthat examples can't). Spec-Driven Development didn't just document my API — it found a bug, made my AI tests free, and became the project's source of truth.\n\nThe real test of \"mock the LLM\" is to run the contract suite against **the endpoints that actually call the LLM** (`/api/command`\n\n) with the provider stubbed — so the AI path is exercised, deterministically, for **zero tokens**. Those endpoints are behind a session cookie, which Specmatic's test mode can't drive, so I added a **gated test-auth** (a Bearer path enabled only when `SPECMATIC_TEST`\n\nis set, never in production) and pointed the app's LLM calls at the stub.\n\nRunning the **full api_contract.yaml** this way immediately earned its keep — it surfaced five real issues I'd never have found by reading the code:\n\n`/api/connections`\n\nleaked an undocumented `config`\n\nobject`/api/command`\n\nerror response was ambiguous`oneOf`\n\n(it carries `sql`\n\n/`task`\n\nalongside `error`\n\n, so it half-looked like a `ReadResult`\n\n). I gave errors their own `CommandError`\n\nschema.`/api/command`\n\nreturned HTTP 500 on a non-string `command`\n\n`None.strip()`\n\ncrashed) — a resiliency negative test caught it; I fixed it to a clean `400`\n\n.`/api/execute`\n\nmishandled malformed `sql`\n\n`400`\n\n.`/api/execute`\n\nhad its body marked requiredTwo of those (3, 4) were genuine **server crashes** on bad input. The whole point landed: *mocking the LLM let the contract suite exercise the AI endpoints for free, and the resiliency tests turned \"we'll find out in production\" into \"we found out in CI.\"*\n\nThe last round was about *signal quality*. Bundling \"does it conform?\" and \"does it survive garbage input?\" into a single green check hides which one broke when it does. So I **split the CI by concern**: for each app spec (`contract_public.yaml`\n\n, `api_contract.yaml`\n\n) there are now two independent jobs — a **contract** job (value-checked conformance against inline + external examples) and a **resiliency** job (generative negative/boundary tests). Two specs × two modes = **four test jobs**, plus a fifth that smoke-tests the LLM virtualization. A conformance regression and a robustness regression now light up as *different* red checks.\n\nI also stopped hand-writing text summaries and let Specmatic emit the real thing: `specmatic.yaml`\n\nconfigures `report.formatters: [text, html]`\n\n, so every job produces a proper **HTML coverage report** (`build/reports/specmatic/test/html`\n\n) — committed under `reports/`\n\nand uploaded as a CI artifact. The numbers are honest per concern: `contract_public`\n\ncontract-mode covers the 200s (50%) while its resiliency job reaches **100%** by exercising the `400`\n\n; `api_contract`\n\nsits at 43% → 64% because the authenticated-only paths and 401s aren't auto-driven by bearer test-mode — the report *shows* that gap rather than papering over it.\n\nFinally, I broadened the **external examples** — the login contract now pins all three roles and the rejection path (`admin1`\n\n→`ADMIN`\n\n, `editor1`\n\n→`EDITOR`\n\n, `viewer1`\n\n→`VIEWER`\n\n, bad creds→`Invalid credentials`\n\n) as separate example files, so the contract job checks real values across the whole auth surface, not just one case.\n\nThe throughline across every round: *the contract isn't documentation you write once — it's an executable artifact you keep sharpening, and each time you sharpen it, it hands you back a bug or a blind spot you didn't know you had.*\n\nThe per-concern reports were honest, and they exposed a real hole: my protected endpoints were only ~43–64% covered. Digging into Specmatic's coverage table, the pattern was clear — every endpoint's `200`\n\nwas tested, but the `400`\n\ns and especially the `401`\n\ns were \"not tested.\" The `401`\n\ns were the interesting ones: my CI test-auth accepted **any** bearer token, and Specmatic always sends one in test mode, so the unauthenticated path was structurally unreachable. My own test shortcut was hiding a whole column of behavior.\n\nThe fix was to make auth **example-driven** instead of blanket: the test-auth gate now accepts exactly one token (the value of `SPECMATIC_TEST`\n\n). Then external examples do the rest — they carry the right token to exercise the authenticated `200`\n\n/`400`\n\nresponses, and a *wrong* token (or none) to exercise the real `401`\n\n. One authenticated run now covers both the authorized and the unauthorized response of every secured endpoint. With examples added for every `400`\n\nand `401`\n\n, all four jobs — contract and resiliency, on both specs — report **100% coverage**. The lesson that stuck: *a convenient test backdoor can quietly suppress coverage; tightening it to be precise is what let the contract test the thing that actually matters (is auth enforced?).*\n\nFinally I migrated `specmatic.yaml`\n\nfrom v2 to **v3**, which replaces v2's implicit `provides`\n\n/`consumes`\n\nwith **explicit service wiring**. That turned out to describe this system almost perfectly: the `systemUnderTest`\n\nis the Meridian Data API (run `type: test`\n\n), and its one external `dependency`\n\n— the LLM provider — is declared as a service run in `type: mock`\n\n. The config now reads like the architecture diagram: *test my API; virtualize the LLM it depends on.* Reports moved under `specmatic.governance.report`\n\n(`formats: [html, ctrf]`\n\n). Six rounds in, the config isn't just settings — it's an honest, executable description of what this service is and what it leans on.\n\nThere's a difference between \"every operation in the contract is tested\" and \"every operation is tested *and actually exists in the running app*.\" Specmatic can prove the second — but only if the app exposes its route table. Spring Boot apps get this free via `/actuator/mappings`\n\n; this is a Flask app, so I built the equivalent: a **test-only /actuator/mappings endpoint** that generates the Spring-Actuator JSON from Flask's own\n\n`url_map`\n\n. Point Specmatic at it (it auto-discovers `/actuator`\n\n), and the \"cannot calculate actual coverage\" warning disappears — the reports now show A reviewer on Python 3.14 hit a wall: `pymssql==2.3.4`\n\nhas no wheel for 3.14 and fails to build. The fix taught a nice lesson about *optional* dependencies. Meridian speaks to eight database engines, but every driver is **lazy-imported** — the app and the entire test suite run on the bundled SQLite databases with none of them installed. So the heavy external drivers didn't belong in the core `requirements.txt`\n\nat all. I split them into `requirements-optional.txt`\n\n(loosely pinned so newer Pythons resolve a compatible wheel) and left the core install lean. Verified by importing the app with every driver blocked: 91 routes, zero errors. A dependency you only need for one optional path shouldn't be able to break everyone's install.\n\nThree small things, each a good reminder that \"works\" and \"configured correctly\" aren't the same:\n\n**The actuator was read but not registered.** My report kept showing the actuator as disabled even though actual coverage was computing. The tell was in the access log: Specmatic was hitting\n\n`/actuator`\n\n(my Spring-style root, which returns `_links`\n\n) but never following to `/actuator/mappings`\n\n. Pointing `actuatorUrl`\n\n**Auth belongs in the config, not scattered across examples.** Following Specmatic's security-schemes pattern, the bearer token is now declared once in `specmatic.yaml`\n\nunder `securitySchemes`\n\n(overridable via `SPECMATIC_BEARER_TOKEN`\n\n), and the one-command test runner starts the app with the matching test-auth token. So both the authenticated paths and the `401`\n\ns are exercised, and there's a single place that owns the credential — no more hunting through fixtures to see how auth works.\n\n** python -m pip, always.** A reviewer on Python 3.14 hit an install-vs-run mismatch that traced back to bare\n\n`pip`\n\nresolving to a different interpreter than the app runs on. The dependency split already made 3.14 install cleanly (verified on 3.11–3.14); documenting `python -m pip`\n\ninside the venv closes the last gap. Small, but it's the difference between \"it works on my machine\" and \"it works on yours.\"The sharpest feedback I got was also the most obvious in hindsight: *your code shouldn't have a separate path for the test tool.* I'd been authenticating Specmatic through a `SPECMATIC_TEST`\n\n-gated bypass — a branch that only existed for contract testing. That's a smell: you're no longer testing the thing you ship. So I removed it and made bearer-token auth a **real, first-class feature** of the API: alongside the web UI's session-cookie login, the API accepts a `Bearer`\n\ntoken (configured via `API_BEARER_TOKEN`\n\n, off by default) that any programmatic client can use — curl, a gateway, CI, or Specmatic. Specmatic just declares that token in `securitySchemes`\n\nand sends it like any other client. The auth path under test is now the *same* path real callers use, and the `401`\n\ns are exercised by simply sending a wrong token. No backdoor, no drift between \"tested\" and \"deployed.\"\n\n**Scope, documented.** With the actuator now honestly reporting the app's 52 routes against a 6-endpoint contract, the report shows 46 \"Missing in Spec.\" Rather than paper over that, I wrote [ CONTRACT_SCOPE.md](https://CONTRACT_SCOPE.md): the six are the trust boundary external consumers depend on (contract-tested at 100%); the rest are the SPA's own feature endpoints, shipped in the same unit, with an explicit promotion path. A contract's job is to guard boundaries, and the gap is now a visible, justified backlog rather than a silent omission.\n\n**And it should just run.** A reviewer hit port conflicts on the fixed test ports. The runner now auto-selects a free port and keeps the app URL and the actuator URL in lock-step via one env var, warms the LLM-mock path before the suite so the first AI scenarios don't race a cold stub, and the README lists the exact per-job commands. \"Clone and run\" shouldn't depend on nothing else living on port 5001.\n\nThe final round of feedback was really about legibility — for the reader, not the tool. Three fixes:\n\n**Two jobs instead of four.** I'd been running a separate \"resiliency mode\" job per spec by toggling the now-deprecated `SPECMATIC_GENERATIVE_TESTS`\n\nenv var true/false. Turns out that's not the current mechanism — Specmatic replaced it with a config field, `schemaResiliencyTests: all`\n\n, set once in `specmatic.yaml`\n\n. With that in place, a *single* `specmatic test`\n\nrun per spec already exercises both conformance (examples) *and* resiliency (generative/boundary) in one pass. So the contract/resiliency split I'd built CI around wasn't actually necessary — it was an artifact of using the deprecated toggle. Down to one job per app spec, same 100% coverage, half the CI jobs.\n\n**One README, not two.** I'd accumulated a full set of run instructions in the middle of the \"Spec-Driven Development\" narrative section, and a *second*, more detailed set in \"Setup\" near the bottom — written at different times, drifted apart, both technically correct but redundant. Consolidated into one canonical section with every configurable variable documented in a table (ports, the bearer token, the jar path, actuator toggle), and the narrative section now just points down to it.\n\n**Scripts that only worked on my machine, gone.** Two local-only helper scripts had a personal absolute path hardcoded into them — exactly the kind of thing that undermines \"clone and run.\" I folded their one useful behavior (regenerating the committed HTML reports) directly into the one script everyone already runs, and deleted them.\n\nThe meta-lesson across the whole engagement, really: almost every round wasn't \"add more\" — it was \"say the same thing in fewer, more honest places.\" That's the same instinct an executable contract enforces on an API: one source of truth, not several that can quietly drift apart.\n\n*Code + integration: github.com/AYON-ARYAN/DATABASE-MANAGER (branches: main + react_build; see api_contract.yaml, SPECMATIC_INTEGRATION.md). Specademy course completed — certificate attached.*\n\n— Ayon Aryan", "url": "https://wpnews.pro/news/executable-contracts-as-guardrails-for-ai-generated-code", "canonical_source": "https://dev.to/ayonaryan/executable-contracts-as-guardrails-for-ai-generated-code-iip", "published_at": "2026-07-13 05:23:37+00:00", "updated_at": "2026-07-13 05:43:57.171136+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-safety", "large-language-models", "artificial-intelligence"], "entities": ["Specmatic", "DATABASE-MANAGER", "Claude Code", "Cursor", "Flask", "React", "OpenAPI"], "alternates": {"html": "https://wpnews.pro/news/executable-contracts-as-guardrails-for-ai-generated-code", "markdown": "https://wpnews.pro/news/executable-contracts-as-guardrails-for-ai-generated-code.md", "text": "https://wpnews.pro/news/executable-contracts-as-guardrails-for-ai-generated-code.txt", "jsonld": "https://wpnews.pro/news/executable-contracts-as-guardrails-for-ai-generated-code.jsonld"}}