{"slug": "i-tested-31-mcp-servers-for-contract-compliance-only-3-passed", "title": "I tested 31 MCP servers for contract compliance. Only 3% passed.", "summary": "A developer built mcp-drill, a fault-injection harness that speaks the Model Context Protocol, and scanned 31 popular MCP servers covering 265 tools including Microsoft Learn, Hugging Face, Cloudflare, and DeepWiki. The scan found that only 3% of tools declare an outputSchema strict enough to reject a corrupted but well-typed response, while 56% declare no schema at all and 42% declare a vacuous schema that validates garbage. The developer argues that with agents chaining tool outputs into tool inputs without human review, a vacuous contract leaves no automatic guard against well-typed but wrong results.", "body_md": "MCP has `outputSchema` so agents can validate tool results. But does the schema actually *reject* a wrong answer? I built [mcp-drill](https://github.com/TimurRakhmatullin86/mcp-drill) -- a fault-injection harness that speaks MCP -- and scanned 31 popular servers (265 tools) including **Microsoft Learn, Hugging Face, Cloudflare, DeepWiki**. Result: **only 3% declare a contract that would reject a corrupted response.** 56% declare *nothing*, 42% declare a schema that happily validates garbage. Your agent cannot tell a bad result from a good one.\n\n`pip install mcp-drill[scan]`\nMCP is JSON-RPC over stdio / Streamable HTTP with bidirectional notifications. Normal HTTP chaos tools don't speak it. And even when you test an MCP server, you usually test *your* agent, not whether the *server's contract* protects you.\n\nReal failures I kept hitting:\n\n`fetch` tool returns `{\"result\": \"ok\"}` with status 200 even when the tool name is wrong -- how does the agent know it failed?`outputSchema: {type: \"object\"}` -- great, it validates I wanted one command to answer: **if I corrupt the response but keep its type, does your schema catch it?** And separately: **if I send you garbage input, do you tell me with a proper error?**\n\nSo I built `mcp-drill`:\n\n`mcp-drill wrap --faults timeout,corrupt,truncate,malformed -- npx ...` -- sits between client and server, perturbs responses deterministically (seed-able).`mcp-drill scan -- npx ...` or `mcp-drill scan --url https://...` -- no LLM, deterministic, reproducible. Every number is a property of the server.\nFor each server, `mcp-drill` does the MCP handshake, lists tools, then runs fixed probes:\n\n`outputSchema` at all.`mcp-drill-corruption` / `-999999999` / out-of-range. If it still validates -> `jsonrpc_error` / `tool_error` (good) vs `accepted` / `timeout` / `crash` (bad).\nRepro: `pip install -e \".[scan]\" && python studies/pilot/run_pilot.py studies/pilot/servers.json` -- commits server list + raw `results.json`.\n\nFull methodology: [METHODOLOGY.md](https://github.com/TimurRakhmatullin86/mcp-drill/blob/main/METHODOLOGY.md)\n\n**31 servers, 265 tools -- 3% enforceable.**\n\n| Tier | Share | Count | Meaning | \n|---|---|---|---|\n| No schema | **56%** | 148/265 | Nothing to validate against | \n| Vacuous schema | **42%** | 110/265 | Corrupted payload still validates -- zero protection | \n| Enforceable | **3%** | 7/265 | Schema rejects the corrupted payload | \n\nNotable:\n\n`x-fastmcp-wrap-result`): 10 tools -- the default from the dominant MCP Python SDK (FastMCP) wraps a return as `{\"result\": string}` and calls it a contract. It's vacuous by construction.\nSome highlights:\n\n| Server | Tools | Enforceable | Notes | \n|---|---|---|---|\n| git-mcp-server | 28 | **18%** | Best of the bunch | \n| huggingface (remote) | 8 | **12%** | Only marquee with enforceable schemas | \n| filesystem | 14 | **7%** | Best reference server | \n| everything | 13 | 0% | Reference server, 100% vacuous | \n| microsoft-learn (remote) | 3 | 0% | Name-brand doesn't help | \n| deepwiki (remote) | 3 | 0% | 100% vacuous | \n| playwright | 23 | 0% | No schemas at all | \n| desktop-commander | 26 | 0% | No schemas at all | \n\nAgents increasingly act on a tool result without a human in the loop: tool A's output becomes tool B's input. The only automatic guard is: *did the transport succeed + did the payload match `outputSchema`?* If the schema is vacuous, nothing guards a well-typed but wrong result, and the agent proceeds on bad data.\n\nThis is not what security scanners (like `mcp-scan`) catch. Those ask \"can this server be abused to do something evil?\" We ask \"can this server be *trusted* when it returns a result?\" See the [VS page](https://timurrakhmatullin86.github.io/mcp-drill/vs-mcp-scan/).\n\nAnd coverage is a vanity metric here. Auto-generated schemas (FastMCP infers from return type hints) raise coverage toward 100% while enforceability stays near 0% -- a vacuous default inherited by every server that doesn't override it. The gap widens as tooling improves, unless schemas add value-level constraints (`enum`, `pattern`, `format`, bounds).\n\n```\n# install\npip install \"mcp-drill[scan]\"\n# or without install\nuvx mcp-drill scan -- --help\n\n# local stdio server\nmcp-drill scan -- npx -y @modelcontextprotocol/server-filesystem /tmp\n\n# remote Streamable HTTP\nmcp-drill scan --url https://mcp.deepwiki.com/mcp\n\n# JSON output for CI\nmcp-drill scan --json -- npx -y @modelcontextprotocol/server-filesystem /tmp > mcp-drill.json\n\n# badge (shields.io endpoint)\nmcp-drill scan --badge --url https://mcp.deepwiki.com/mcp > badge.json\n\n# fault injection proxy\nmcp-drill wrap --faults timeout,truncate -- npx -y @modelcontextprotocol/server-everything\n```\n\nGate in CI -- GitHub Action (no LLM, no API key):\n\n```\n- uses: TimurRakhmatullin86/mcp-drill@v0\n  with:\n    server: 'npx -y @modelcontextprotocol/server-filesystem /tmp'\n    min-error-handling: '0.9'\n```\n\n`enum` / `pattern` / `format` / numeric bounds where semantics allow. `additionalProperties: false` helps, but alone it's not enough -- a corrupted string is still a string. Test with `mcp-drill scan --json` in CI and gate on it.`mcp-drill wrap` to exercise your agent's failure paths before prod.\n[GitHub repo](https://github.com/TimurRakhmatullin86/mcp-drill) -- Apache-2.0, telemetry off, Python 3.10+. [Live scorecard](https://timurrakhmatullin86.github.io/mcp-drill/). PRs and issues welcome -- especially if your server scores differently and you think the harness is wrong.\n\n*Method is model-free and deterministic -- every number is a property of the server, not of whatever agent called it. The tool is the methodology, and it is released.*", "url": "https://wpnews.pro/news/i-tested-31-mcp-servers-for-contract-compliance-only-3-passed", "canonical_source": "https://dev.to/tim860/i-tested-31-mcp-servers-for-contract-compliance-only-3-passed-25gp", "published_at": "2026-09-13 04:25:02+00:00", "updated_at": "2026-09-13 04:56:35.229164+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety", "mlops"], "entities": ["mcp-drill", "Model Context Protocol", "Microsoft Learn", "Hugging Face", "Cloudflare", "DeepWiki", "FastMCP", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/i-tested-31-mcp-servers-for-contract-compliance-only-3-passed", "markdown": "https://wpnews.pro/news/i-tested-31-mcp-servers-for-contract-compliance-only-3-passed.md", "text": "https://wpnews.pro/news/i-tested-31-mcp-servers-for-contract-compliance-only-3-passed.txt", "jsonld": "https://wpnews.pro/news/i-tested-31-mcp-servers-for-contract-compliance-only-3-passed.jsonld"}}