I tested 31 MCP servers for contract compliance. Only 3% passed.

A developer built mcp-drill, a fault-injection harness that speaks the Model Context Protocol, and scanned 31 popular MCP servers covering 265 tools including Microsoft Learn, Hugging Face, Cloudflare, and DeepWiki. The scan found that only 3% of tools declare an outputSchema strict enough to reject a corrupted but well-typed response, while 56% declare no schema at all and 42% declare a vacuous schema that validates garbage. The developer argues that with agents chaining tool outputs into tool inputs without human review, a vacuous contract leaves no automatic guard against well-typed but wrong results.

MCP has outputSchema so agents can validate tool results. But does the schema actually reject a wrong answer? I built mcp-drill https://github.com/TimurRakhmatullin86/mcp-drill -- a fault-injection harness that speaks MCP -- and scanned 31 popular servers 265 tools including Microsoft Learn, Hugging Face, Cloudflare, DeepWiki . Result: only 3% declare a contract that would reject a corrupted response. 56% declare nothing , 42% declare a schema that happily validates garbage. Your agent cannot tell a bad result from a good one. pip install mcp-drill scan MCP is JSON-RPC over stdio / Streamable HTTP with bidirectional notifications. Normal HTTP chaos tools don't speak it. And even when you test an MCP server, you usually test your agent, not whether the server's contract protects you. Real failures I kept hitting: fetch tool returns {"result": "ok"} with status 200 even when the tool name is wrong -- how does the agent know it failed? outputSchema: {type: "object"} -- great, it validates I wanted one command to answer: if I corrupt the response but keep its type, does your schema catch it? And separately: if I send you garbage input, do you tell me with a proper error? So I built mcp-drill : mcp-drill wrap --faults timeout,corrupt,truncate,malformed -- npx ... -- sits between client and server, perturbs responses deterministically seed-able . mcp-drill scan -- npx ... or mcp-drill scan --url https://... -- no LLM, deterministic, reproducible. Every number is a property of the server. For each server, mcp-drill does the MCP handshake, lists tools, then runs fixed probes: outputSchema at all. mcp-drill-corruption / -999999999 / out-of-range. If it still validates - jsonrpc error / tool error good vs accepted / timeout / crash bad . Repro: pip install -e ". scan " && python studies/pilot/run pilot.py studies/pilot/servers.json -- commits server list + raw results.json . Full methodology: METHODOLOGY.md https://github.com/TimurRakhmatullin86/mcp-drill/blob/main/METHODOLOGY.md 31 servers, 265 tools -- 3% enforceable. | Tier | Share | Count | Meaning | |---|---|---|---| | No schema | 56% | 148/265 | Nothing to validate against | | Vacuous schema | 42% | 110/265 | Corrupted payload still validates -- zero protection | | Enforceable | 3% | 7/265 | Schema rejects the corrupted payload | Notable: x-fastmcp-wrap-result : 10 tools -- the default from the dominant MCP Python SDK FastMCP wraps a return as {"result": string} and calls it a contract. It's vacuous by construction. Some highlights: | Server | Tools | Enforceable | Notes | |---|---|---|---| | git-mcp-server | 28 | 18% | Best of the bunch | | huggingface remote | 8 | 12% | Only marquee with enforceable schemas | | filesystem | 14 | 7% | Best reference server | | everything | 13 | 0% | Reference server, 100% vacuous | | microsoft-learn remote | 3 | 0% | Name-brand doesn't help | | deepwiki remote | 3 | 0% | 100% vacuous | | playwright | 23 | 0% | No schemas at all | | desktop-commander | 26 | 0% | No schemas at all | Agents increasingly act on a tool result without a human in the loop: tool A's output becomes tool B's input. The only automatic guard is: did the transport succeed + did the payload match outputSchema ? If the schema is vacuous, nothing guards a well-typed but wrong result, and the agent proceeds on bad data. This is not what security scanners like mcp-scan catch. Those ask "can this server be abused to do something evil?" We ask "can this server be trusted when it returns a result?" See the VS page https://timurrakhmatullin86.github.io/mcp-drill/vs-mcp-scan/ . And coverage is a vanity metric here. Auto-generated schemas FastMCP infers from return type hints raise coverage toward 100% while enforceability stays near 0% -- a vacuous default inherited by every server that doesn't override it. The gap widens as tooling improves, unless schemas add value-level constraints enum , pattern , format , bounds . install pip install "mcp-drill scan " or without install uvx mcp-drill scan -- --help local stdio server mcp-drill scan -- npx -y @modelcontextprotocol/server-filesystem /tmp remote Streamable HTTP mcp-drill scan --url https://mcp.deepwiki.com/mcp JSON output for CI mcp-drill scan --json -- npx -y @modelcontextprotocol/server-filesystem /tmp mcp-drill.json badge shields.io endpoint mcp-drill scan --badge --url https://mcp.deepwiki.com/mcp badge.json fault injection proxy mcp-drill wrap --faults timeout,truncate -- npx -y @modelcontextprotocol/server-everything Gate in CI -- GitHub Action no LLM, no API key : - uses: TimurRakhmatullin86/mcp-drill@v0 with: server: 'npx -y @modelcontextprotocol/server-filesystem /tmp' min-error-handling: '0.9' enum / pattern / format / numeric bounds where semantics allow. additionalProperties: false helps, but alone it's not enough -- a corrupted string is still a string. Test with mcp-drill scan --json in CI and gate on it. mcp-drill wrap to exercise your agent's failure paths before prod. GitHub repo https://github.com/TimurRakhmatullin86/mcp-drill -- Apache-2.0, telemetry off, Python 3.10+. Live scorecard https://timurrakhmatullin86.github.io/mcp-drill/ . PRs and issues welcome -- especially if your server scores differently and you think the harness is wrong. Method is model-free and deterministic -- every number is a property of the server, not of whatever agent called it. The tool is the methodology, and it is released.