# Executable Contracts as Guardrails for AI-Generated Code

> Source: <https://dev.to/ayonaryan/executable-contracts-as-guardrails-for-ai-generated-code-iip>
> Published: 2026-07-13 05:23:37+00:00

*Submitted for the Specmatic Full Stack AI Engineering Intern challenge.*

I build with AI coding agents every day — Claude Code, Cursor. They're incredible at velocity. But they have one quiet, expensive failure mode: **they drift.** You ask for a small change to an endpoint, and somewhere in the diff the agent renames a field, flips a status code, or drops a required parameter. The code still runs. The tests (if you have them) still pass. And then three layers downstream — in the frontend, in a consumer service — something breaks, and you spend an afternoon debugging an integration bug that was never a logic bug at all. It was a **contract** bug.

That's the exact problem Specmatic's challenge asked me to explore: *can Spec-Driven Development and executable contracts improve AI-assisted software development?* After integrating Specmatic into one of my projects, my answer is an emphatic yes — and the reason is that **an executable contract is a guardrail for AI-generated code.**

[DATABASE-MANAGER](https://github.com/AYON-ARYAN/DATABASE-MANAGER) is a natural-language-to-SQL engine I built that works across 8 database engines (SQLite, MySQL, PostgreSQL, MSSQL, Oracle, MongoDB, Cassandra, Redis). It has a Flask `/api`

blueprint consumed by a React frontend, and — importantly — a **human-in-the-loop write-safety model**: any write/schema command doesn't execute directly. The API returns `{ "needs_review": true, ... }`

, a human approves, and only then does `/api/execute`

run it, with snapshot rollback via `/api/undo`

.

So the project already had a philosophy: *don't let the model run unchecked.* Specmatic let me extend that exact philosophy from **database writes** to the **API contract itself.**

I wrote an OpenAPI contract (`api_contract.yaml`

) describing the real API. The interesting part is the `/api/command`

response, which is a *union* of three shapes — a READ result, a write staged for review, or an error:

```
/api/command:
  post:
    responses:
      "200":
        content:
          application/json:
            schema:
              oneOf:
                - $ref: "#/components/schemas/ReadResult"
                - $ref: "#/components/schemas/NeedsReview"   # { needs_review: true, sql, explanation, task }
                - $ref: "#/components/schemas/Error"
```

This contract is now the single thing both the backend and the frontend agree on.

With Specmatic, the contract isn't documentation that rots — it's *executable*. Specmatic auto-generates positive and negative requests from the spec and verifies the live API conforms:

```
docker run --rm --network host -v "$PWD:/specs" -w /specs \
  specmatic/specmatic:latest test --host localhost --port 5001
```

No hand-written test cases. The spec *is* the test suite.

Specmatic can also *be* the API — a spec-conformant stub — so my React frontend can develop with no backend, no database, and no LLM keys running:

```
docker run --rm --network host -v "$PWD:/specs" -w /specs \
  specmatic/specmatic:latest stub --port 9000
```

Same contract, two uses: it tests the provider *and* mocks it for the consumer. That's contract-driven development.

Here's the demo that made it click for me. I asked an AI agent to "add a confidence score and rename `needs_review`

to `requiresReview`

." It did — silently changing the response shape. To a human reviewer skimming the diff, it looks harmless. To the React frontend, it's a broken integration.

Then I re-ran the contract test:

**Specmatic caught it instantly** and told me *exactly* what drifted. A bug that would have surfaced as a confusing frontend error became a precise, one-line test failure — before it ever left my machine.

Building with LLMs taught me a mantra: *it's 20% prompting and 80% guardrails.* The interesting engineering is everything you put **around** the model to make it safe. Specmatic is that idea applied to AI-assisted development itself:

For a tool like DATABASE-MANAGER — where I already gate database writes behind human review — adding a contract guardrail at the API layer felt like the natural second half of the same idea. Two guardrails, one principle: don't let generated actions run unchecked.

That's how Spec-Driven Development improves AI-assisted software engineering. It turns the AI's biggest weakness — confident, silent drift — into the cheapest possible signal: a failing test.

After the first pass I pushed three Specmatic techniques further:

**1. Inline + external examples.** Instead of letting Specmatic only check *shapes*, I added examples that pin *values* — so the contract test now verifies real cases: `admin1`

→`ADMIN`

, bad creds→`Invalid credentials`

, and `editor1`

→`EDITOR`

(the last loaded as an external example file). One generic test became three value-checked ones, and the stub now returns lifelike data.

**2. Schema-resiliency testing found a real bug.** Turning on generative negative tests, Specmatic mutated the login request — empty body, `username`

as null / number / boolean — and expected a `4xx`

. My API returned `200`

for all of them: it had **no input validation**, silently treating a malformed body as a failed login. 14 negative tests failed. I added body validation (`400`

on a non-object body or non-string fields), documented the `400`

, and re-ran: **24/24 green**. The resiliency suite turned an invisible robustness gap into a fixed, tested guarantee.

**3. Virtualizing the LLM itself.** Meridian's NL-to-SQL calls an LLM (Groq, which is OpenAI-compatible). Real tests would burn tokens on every CI run and tolerate non-determinism. So I wrote a contract for the LLM's `/v1/chat/completions`

, ran `specmatic stub`

, and pointed the app at it with a single env var.

*How it's wired:* `core/llm_manager.py`

reads `GROQ_API_URL`

(unset in prod → real Groq); in CI a step boots `specmatic stub llm_contract.yaml`

, sets `GROQ_API_URL`

to it, and runs the real NL-to-SQL path against the stub — so the AI path is **offline, deterministic, and zero-token**. It's a *separate* CI step, after the API contract tests, because it virtualizes an upstream *dependency* the app **consumes**, not the app's own API. The stub spec is a deliberately reduced version of the real provider API — every deviation (and why) is documented in `LLM_CONTRACT_NOTES.md`

. The same tool that guards *my* API also virtualizes the *dependency* my API consumes — exactly the kind of guardrail an AI-native codebase needs.

All three run in CI on every push, and the contract test reports **100% coverage** (generative negative tests cover the `400`

that examples can't). Spec-Driven Development didn't just document my API — it found a bug, made my AI tests free, and became the project's source of truth.

The real test of "mock the LLM" is to run the contract suite against **the endpoints that actually call the LLM** (`/api/command`

) with the provider stubbed — so the AI path is exercised, deterministically, for **zero tokens**. Those endpoints are behind a session cookie, which Specmatic's test mode can't drive, so I added a **gated test-auth** (a Bearer path enabled only when `SPECMATIC_TEST`

is set, never in production) and pointed the app's LLM calls at the stub.

Running the **full api_contract.yaml** this way immediately earned its keep — it surfaced five real issues I'd never have found by reading the code:

`/api/connections`

leaked an undocumented `config`

object`/api/command`

error response was ambiguous`oneOf`

(it carries `sql`

/`task`

alongside `error`

, so it half-looked like a `ReadResult`

). I gave errors their own `CommandError`

schema.`/api/command`

returned HTTP 500 on a non-string `command`

`None.strip()`

crashed) — a resiliency negative test caught it; I fixed it to a clean `400`

.`/api/execute`

mishandled malformed `sql`

`400`

.`/api/execute`

had its body marked requiredTwo of those (3, 4) were genuine **server crashes** on bad input. The whole point landed: *mocking the LLM let the contract suite exercise the AI endpoints for free, and the resiliency tests turned "we'll find out in production" into "we found out in CI."*

The last round was about *signal quality*. Bundling "does it conform?" and "does it survive garbage input?" into a single green check hides which one broke when it does. So I **split the CI by concern**: for each app spec (`contract_public.yaml`

, `api_contract.yaml`

) there are now two independent jobs — a **contract** job (value-checked conformance against inline + external examples) and a **resiliency** job (generative negative/boundary tests). Two specs × two modes = **four test jobs**, plus a fifth that smoke-tests the LLM virtualization. A conformance regression and a robustness regression now light up as *different* red checks.

I also stopped hand-writing text summaries and let Specmatic emit the real thing: `specmatic.yaml`

configures `report.formatters: [text, html]`

, so every job produces a proper **HTML coverage report** (`build/reports/specmatic/test/html`

) — committed under `reports/`

and uploaded as a CI artifact. The numbers are honest per concern: `contract_public`

contract-mode covers the 200s (50%) while its resiliency job reaches **100%** by exercising the `400`

; `api_contract`

sits at 43% → 64% because the authenticated-only paths and 401s aren't auto-driven by bearer test-mode — the report *shows* that gap rather than papering over it.

Finally, I broadened the **external examples** — the login contract now pins all three roles and the rejection path (`admin1`

→`ADMIN`

, `editor1`

→`EDITOR`

, `viewer1`

→`VIEWER`

, bad creds→`Invalid credentials`

) as separate example files, so the contract job checks real values across the whole auth surface, not just one case.

The throughline across every round: *the contract isn't documentation you write once — it's an executable artifact you keep sharpening, and each time you sharpen it, it hands you back a bug or a blind spot you didn't know you had.*

The per-concern reports were honest, and they exposed a real hole: my protected endpoints were only ~43–64% covered. Digging into Specmatic's coverage table, the pattern was clear — every endpoint's `200`

was tested, but the `400`

s and especially the `401`

s were "not tested." The `401`

s were the interesting ones: my CI test-auth accepted **any** bearer token, and Specmatic always sends one in test mode, so the unauthenticated path was structurally unreachable. My own test shortcut was hiding a whole column of behavior.

The fix was to make auth **example-driven** instead of blanket: the test-auth gate now accepts exactly one token (the value of `SPECMATIC_TEST`

). Then external examples do the rest — they carry the right token to exercise the authenticated `200`

/`400`

responses, and a *wrong* token (or none) to exercise the real `401`

. One authenticated run now covers both the authorized and the unauthorized response of every secured endpoint. With examples added for every `400`

and `401`

, all four jobs — contract and resiliency, on both specs — report **100% coverage**. The lesson that stuck: *a convenient test backdoor can quietly suppress coverage; tightening it to be precise is what let the contract test the thing that actually matters (is auth enforced?).*

Finally I migrated `specmatic.yaml`

from v2 to **v3**, which replaces v2's implicit `provides`

/`consumes`

with **explicit service wiring**. That turned out to describe this system almost perfectly: the `systemUnderTest`

is the Meridian Data API (run `type: test`

), and its one external `dependency`

— the LLM provider — is declared as a service run in `type: mock`

. The config now reads like the architecture diagram: *test my API; virtualize the LLM it depends on.* Reports moved under `specmatic.governance.report`

(`formats: [html, ctrf]`

). Six rounds in, the config isn't just settings — it's an honest, executable description of what this service is and what it leans on.

There's a difference between "every operation in the contract is tested" and "every operation is tested *and actually exists in the running app*." Specmatic can prove the second — but only if the app exposes its route table. Spring Boot apps get this free via `/actuator/mappings`

; this is a Flask app, so I built the equivalent: a **test-only /actuator/mappings endpoint** that generates the Spring-Actuator JSON from Flask's own

`url_map`

. Point Specmatic at it (it auto-discovers `/actuator`

), and the "cannot calculate actual coverage" warning disappears — the reports now show A reviewer on Python 3.14 hit a wall: `pymssql==2.3.4`

has no wheel for 3.14 and fails to build. The fix taught a nice lesson about *optional* dependencies. Meridian speaks to eight database engines, but every driver is **lazy-imported** — the app and the entire test suite run on the bundled SQLite databases with none of them installed. So the heavy external drivers didn't belong in the core `requirements.txt`

at all. I split them into `requirements-optional.txt`

(loosely pinned so newer Pythons resolve a compatible wheel) and left the core install lean. Verified by importing the app with every driver blocked: 91 routes, zero errors. A dependency you only need for one optional path shouldn't be able to break everyone's install.

Three small things, each a good reminder that "works" and "configured correctly" aren't the same:

**The actuator was read but not registered.** My report kept showing the actuator as disabled even though actual coverage was computing. The tell was in the access log: Specmatic was hitting

`/actuator`

(my Spring-style root, which returns `_links`

) but never following to `/actuator/mappings`

. Pointing `actuatorUrl`

**Auth belongs in the config, not scattered across examples.** Following Specmatic's security-schemes pattern, the bearer token is now declared once in `specmatic.yaml`

under `securitySchemes`

(overridable via `SPECMATIC_BEARER_TOKEN`

), and the one-command test runner starts the app with the matching test-auth token. So both the authenticated paths and the `401`

s are exercised, and there's a single place that owns the credential — no more hunting through fixtures to see how auth works.

** python -m pip, always.** A reviewer on Python 3.14 hit an install-vs-run mismatch that traced back to bare

`pip`

resolving to a different interpreter than the app runs on. The dependency split already made 3.14 install cleanly (verified on 3.11–3.14); documenting `python -m pip`

inside the venv closes the last gap. Small, but it's the difference between "it works on my machine" and "it works on yours."The sharpest feedback I got was also the most obvious in hindsight: *your code shouldn't have a separate path for the test tool.* I'd been authenticating Specmatic through a `SPECMATIC_TEST`

-gated bypass — a branch that only existed for contract testing. That's a smell: you're no longer testing the thing you ship. So I removed it and made bearer-token auth a **real, first-class feature** of the API: alongside the web UI's session-cookie login, the API accepts a `Bearer`

token (configured via `API_BEARER_TOKEN`

, off by default) that any programmatic client can use — curl, a gateway, CI, or Specmatic. Specmatic just declares that token in `securitySchemes`

and sends it like any other client. The auth path under test is now the *same* path real callers use, and the `401`

s are exercised by simply sending a wrong token. No backdoor, no drift between "tested" and "deployed."

**Scope, documented.** With the actuator now honestly reporting the app's 52 routes against a 6-endpoint contract, the report shows 46 "Missing in Spec." Rather than paper over that, I wrote [ CONTRACT_SCOPE.md](https://CONTRACT_SCOPE.md): the six are the trust boundary external consumers depend on (contract-tested at 100%); the rest are the SPA's own feature endpoints, shipped in the same unit, with an explicit promotion path. A contract's job is to guard boundaries, and the gap is now a visible, justified backlog rather than a silent omission.

**And it should just run.** A reviewer hit port conflicts on the fixed test ports. The runner now auto-selects a free port and keeps the app URL and the actuator URL in lock-step via one env var, warms the LLM-mock path before the suite so the first AI scenarios don't race a cold stub, and the README lists the exact per-job commands. "Clone and run" shouldn't depend on nothing else living on port 5001.

The final round of feedback was really about legibility — for the reader, not the tool. Three fixes:

**Two jobs instead of four.** I'd been running a separate "resiliency mode" job per spec by toggling the now-deprecated `SPECMATIC_GENERATIVE_TESTS`

env var true/false. Turns out that's not the current mechanism — Specmatic replaced it with a config field, `schemaResiliencyTests: all`

, set once in `specmatic.yaml`

. With that in place, a *single* `specmatic test`

run per spec already exercises both conformance (examples) *and* resiliency (generative/boundary) in one pass. So the contract/resiliency split I'd built CI around wasn't actually necessary — it was an artifact of using the deprecated toggle. Down to one job per app spec, same 100% coverage, half the CI jobs.

**One README, not two.** I'd accumulated a full set of run instructions in the middle of the "Spec-Driven Development" narrative section, and a *second*, more detailed set in "Setup" near the bottom — written at different times, drifted apart, both technically correct but redundant. Consolidated into one canonical section with every configurable variable documented in a table (ports, the bearer token, the jar path, actuator toggle), and the narrative section now just points down to it.

**Scripts that only worked on my machine, gone.** Two local-only helper scripts had a personal absolute path hardcoded into them — exactly the kind of thing that undermines "clone and run." I folded their one useful behavior (regenerating the committed HTML reports) directly into the one script everyone already runs, and deleted them.

The meta-lesson across the whole engagement, really: almost every round wasn't "add more" — it was "say the same thing in fewer, more honest places." That's the same instinct an executable contract enforces on an API: one source of truth, not several that can quietly drift apart.

*Code + integration: github.com/AYON-ARYAN/DATABASE-MANAGER (branches: main + react_build; see api_contract.yaml, SPECMATIC_INTEGRATION.md). Specademy course completed — certificate attached.*

— Ayon Aryan