MCP has outputSchema so agents can validate tool results. But does the schema actually reject a wrong answer? I built mcp-drill -- a fault-injection harness that speaks MCP -- and scanned 31 popular servers (265 tools) including Microsoft Learn, Hugging Face, Cloudflare, DeepWiki. Result: only 3% declare a contract that would reject a corrupted response. 56% declare nothing, 42% declare a schema that happily validates garbage. Your agent cannot tell a bad result from a good one.
pip install mcp-drill[scan]
MCP is JSON-RPC over stdio / Streamable HTTP with bidirectional notifications. Normal HTTP chaos tools don't speak it. And even when you test an MCP server, you usually test your agent, not whether the server's contract protects you.
Real failures I kept hitting:
fetch tool returns {"result": "ok"} with status 200 even when the tool name is wrong -- how does the agent know it failed?outputSchema: {type: "object"} -- great, it validates I wanted one command to answer: if I corrupt the response but keep its type, does your schema catch it? And separately: if I send you garbage input, do you tell me with a proper error?
So I built mcp-drill:
mcp-drill wrap --faults timeout,corrupt,truncate,malformed -- npx ... -- sits between client and server, perturbs responses deterministically (seed-able).mcp-drill scan -- npx ... or mcp-drill scan --url https://... -- no LLM, deterministic, reproducible. Every number is a property of the server.
For each server, mcp-drill does the MCP handshake, lists tools, then runs fixed probes:
outputSchema at all.mcp-drill-corruption / -999999999 / out-of-range. If it still validates -> jsonrpc_error / tool_error (good) vs accepted / timeout / crash (bad).
Repro: pip install -e ".[scan]" && python studies/pilot/run_pilot.py studies/pilot/servers.json -- commits server list + raw results.json.
Full methodology: METHODOLOGY.md
31 servers, 265 tools -- 3% enforceable.
| Tier | Share | Count | Meaning |
|---|---|---|---|
| No schema | 56% | 148/265 | Nothing to validate against |
| Vacuous schema | 42% | 110/265 | Corrupted payload still validates -- zero protection |
| Enforceable | 3% | 7/265 | Schema rejects the corrupted payload |
Notable:
x-fastmcp-wrap-result): 10 tools -- the default from the dominant MCP Python SDK (FastMCP) wraps a return as {"result": string} and calls it a contract. It's vacuous by construction.
Some highlights:
| Server | Tools | Enforceable | Notes |
|---|---|---|---|
| git-mcp-server | 28 | 18% | Best of the bunch |
| huggingface (remote) | 8 | 12% | Only marquee with enforceable schemas |
| filesystem | 14 | 7% | Best reference server |
| everything | 13 | 0% | Reference server, 100% vacuous |
| microsoft-learn (remote) | 3 | 0% | Name-brand doesn't help |
| deepwiki (remote) | 3 | 0% | 100% vacuous |
| playwright | 23 | 0% | No schemas at all |
| desktop-commander | 26 | 0% | No schemas at all |
Agents increasingly act on a tool result without a human in the loop: tool A's output becomes tool B's input. The only automatic guard is: did the transport succeed + did the payload match outputSchema? If the schema is vacuous, nothing guards a well-typed but wrong result, and the agent proceeds on bad data.
This is not what security scanners (like mcp-scan) catch. Those ask "can this server be abused to do something evil?" We ask "can this server be trusted when it returns a result?" See the VS page.
And coverage is a vanity metric here. Auto-generated schemas (FastMCP infers from return type hints) raise coverage toward 100% while enforceability stays near 0% -- a vacuous default inherited by every server that doesn't override it. The gap widens as tooling improves, unless schemas add value-level constraints (enum, pattern, format, bounds).
pip install "mcp-drill[scan]"
uvx mcp-drill scan -- --help
mcp-drill scan -- npx -y @modelcontextprotocol/server-filesystem /tmp
mcp-drill scan --url https://mcp.deepwiki.com/mcp
mcp-drill scan --json -- npx -y @modelcontextprotocol/server-filesystem /tmp > mcp-drill.json
mcp-drill scan --badge --url https://mcp.deepwiki.com/mcp > badge.json
mcp-drill wrap --faults timeout,truncate -- npx -y @modelcontextprotocol/server-everything
Gate in CI -- GitHub Action (no LLM, no API key):
- uses: TimurRakhmatullin86/mcp-drill@v0
with:
server: 'npx -y @modelcontextprotocol/server-filesystem /tmp'
min-error-handling: '0.9'
enum / pattern / format / numeric bounds where semantics allow. additionalProperties: false helps, but alone it's not enough -- a corrupted string is still a string. Test with mcp-drill scan --json in CI and gate on it.mcp-drill wrap to exercise your agent's failure paths before prod.
GitHub repo -- Apache-2.0, telemetry off, Python 3.10+. Live scorecard. PRs and issues welcome -- especially if your server scores differently and you think the harness is wrong.
Method is model-free and deterministic -- every number is a property of the server, not of whatever agent called it. The tool is the methodology, and it is released.