# I tested 31 MCP servers for contract compliance. Only 3% passed.

> Source: <https://dev.to/tim860/i-tested-31-mcp-servers-for-contract-compliance-only-3-passed-25gp>
> Published: 2026-09-13 04:25:02+00:00

MCP has `outputSchema` so agents can validate tool results. But does the schema actually *reject* a wrong answer? I built [mcp-drill](https://github.com/TimurRakhmatullin86/mcp-drill) -- a fault-injection harness that speaks MCP -- and scanned 31 popular servers (265 tools) including **Microsoft Learn, Hugging Face, Cloudflare, DeepWiki**. Result: **only 3% declare a contract that would reject a corrupted response.** 56% declare *nothing*, 42% declare a schema that happily validates garbage. Your agent cannot tell a bad result from a good one.

`pip install mcp-drill[scan]`
MCP is JSON-RPC over stdio / Streamable HTTP with bidirectional notifications. Normal HTTP chaos tools don't speak it. And even when you test an MCP server, you usually test *your* agent, not whether the *server's contract* protects you.

Real failures I kept hitting:

`fetch` tool returns `{"result": "ok"}` with status 200 even when the tool name is wrong -- how does the agent know it failed?`outputSchema: {type: "object"}` -- great, it validates I wanted one command to answer: **if I corrupt the response but keep its type, does your schema catch it?** And separately: **if I send you garbage input, do you tell me with a proper error?**

So I built `mcp-drill`:

`mcp-drill wrap --faults timeout,corrupt,truncate,malformed -- npx ...` -- sits between client and server, perturbs responses deterministically (seed-able).`mcp-drill scan -- npx ...` or `mcp-drill scan --url https://...` -- no LLM, deterministic, reproducible. Every number is a property of the server.
For each server, `mcp-drill` does the MCP handshake, lists tools, then runs fixed probes:

`outputSchema` at all.`mcp-drill-corruption` / `-999999999` / out-of-range. If it still validates -> `jsonrpc_error` / `tool_error` (good) vs `accepted` / `timeout` / `crash` (bad).
Repro: `pip install -e ".[scan]" && python studies/pilot/run_pilot.py studies/pilot/servers.json` -- commits server list + raw `results.json`.

Full methodology: [METHODOLOGY.md](https://github.com/TimurRakhmatullin86/mcp-drill/blob/main/METHODOLOGY.md)

**31 servers, 265 tools -- 3% enforceable.**

| Tier | Share | Count | Meaning | 
|---|---|---|---|
| No schema | **56%** | 148/265 | Nothing to validate against | 
| Vacuous schema | **42%** | 110/265 | Corrupted payload still validates -- zero protection | 
| Enforceable | **3%** | 7/265 | Schema rejects the corrupted payload | 

Notable:

`x-fastmcp-wrap-result`): 10 tools -- the default from the dominant MCP Python SDK (FastMCP) wraps a return as `{"result": string}` and calls it a contract. It's vacuous by construction.
Some highlights:

| Server | Tools | Enforceable | Notes | 
|---|---|---|---|
| git-mcp-server | 28 | **18%** | Best of the bunch | 
| huggingface (remote) | 8 | **12%** | Only marquee with enforceable schemas | 
| filesystem | 14 | **7%** | Best reference server | 
| everything | 13 | 0% | Reference server, 100% vacuous | 
| microsoft-learn (remote) | 3 | 0% | Name-brand doesn't help | 
| deepwiki (remote) | 3 | 0% | 100% vacuous | 
| playwright | 23 | 0% | No schemas at all | 
| desktop-commander | 26 | 0% | No schemas at all | 

Agents increasingly act on a tool result without a human in the loop: tool A's output becomes tool B's input. The only automatic guard is: *did the transport succeed + did the payload match `outputSchema`?* If the schema is vacuous, nothing guards a well-typed but wrong result, and the agent proceeds on bad data.

This is not what security scanners (like `mcp-scan`) catch. Those ask "can this server be abused to do something evil?" We ask "can this server be *trusted* when it returns a result?" See the [VS page](https://timurrakhmatullin86.github.io/mcp-drill/vs-mcp-scan/).

And coverage is a vanity metric here. Auto-generated schemas (FastMCP infers from return type hints) raise coverage toward 100% while enforceability stays near 0% -- a vacuous default inherited by every server that doesn't override it. The gap widens as tooling improves, unless schemas add value-level constraints (`enum`, `pattern`, `format`, bounds).

```
# install
pip install "mcp-drill[scan]"
# or without install
uvx mcp-drill scan -- --help

# local stdio server
mcp-drill scan -- npx -y @modelcontextprotocol/server-filesystem /tmp

# remote Streamable HTTP
mcp-drill scan --url https://mcp.deepwiki.com/mcp

# JSON output for CI
mcp-drill scan --json -- npx -y @modelcontextprotocol/server-filesystem /tmp > mcp-drill.json

# badge (shields.io endpoint)
mcp-drill scan --badge --url https://mcp.deepwiki.com/mcp > badge.json

# fault injection proxy
mcp-drill wrap --faults timeout,truncate -- npx -y @modelcontextprotocol/server-everything
```

Gate in CI -- GitHub Action (no LLM, no API key):

```
- uses: TimurRakhmatullin86/mcp-drill@v0
  with:
    server: 'npx -y @modelcontextprotocol/server-filesystem /tmp'
    min-error-handling: '0.9'
```

`enum` / `pattern` / `format` / numeric bounds where semantics allow. `additionalProperties: false` helps, but alone it's not enough -- a corrupted string is still a string. Test with `mcp-drill scan --json` in CI and gate on it.`mcp-drill wrap` to exercise your agent's failure paths before prod.
[GitHub repo](https://github.com/TimurRakhmatullin86/mcp-drill) -- Apache-2.0, telemetry off, Python 3.10+. [Live scorecard](https://timurrakhmatullin86.github.io/mcp-drill/). PRs and issues welcome -- especially if your server scores differently and you think the harness is wrong.

*Method is model-free and deterministic -- every number is a property of the server, not of whatever agent called it. The tool is the methodology, and it is released.*