{"slug": "i-built-an-open-source-tool-to-test-what-happens-when-mcp-servers-fail", "title": "I Built an Open-Source Tool to Test What Happens When MCP Servers Fail", "summary": "A developer released MCP Verdict, an open-source Python CLI on PyPI that tests whether Model Context Protocol servers report failures correctly. Developers define expected server behavior in YAML — such as whether a tool should return an error or success — and the tool connects to the server, runs the tool, and flags mismatches, like a tool returning \"Operation failed\" while setting isError to false. Version 0.1.0 requires Python 3.11 or 3.12 and ships with calculator, error-handling, and local HTTP security examples.", "body_md": "I've been spending a lot of time exploring MCP servers and AI agents lately.\n\nOne thing that caught my attention: **we spend a lot of time testing whether tools work, but what about when they don't?**\n\nWhat happens when an MCP tool crashes? When it receives invalid arguments? When a protected endpoint receives an unauthorized request?\n\nAnd more importantly, does the server actually report those failures correctly?\n\nThat's what led me to build **MCP Verdict**.\n\nImagine an MCP tool fails internally but returns something like this:\n\n```\n{\n  \"isError\": false,\n  \"content\": [\n    {\n      \"type\": \"text\",\n      \"text\": \"Operation failed\"\n    }\n  ]\n}\n```\n\nThe message says the operation failed, but the MCP response reports success.\n\nThat inconsistency can cause problems for clients and agents relying on the tool's result.\n\nAnd it's exactly the kind of behavior that's worth testing before deploying an MCP server.\n\n[MCP Verdict](https://github.com/kuruash/mcp-verdict) is an open-source Python CLI that lets developers define expected MCP server behavior and check what actually happens.\n\nInstead of writing separate scripts for every failure scenario, you describe your expectations in YAML.\n\nFor example:\n\n```\nserver:\n  command:\n    - python\n    - examples/basic_stdio/server.py\n\ntool:\n  name: ping\n  arguments: {}\n\nexpected: success\n```\n\nThen run:\n\n```\nmcp-verdict check --config verdict.yaml\n```\n\nMCP Verdict connects to the server, executes the tool, and checks whether the actual result matches your expectation.\n\nThe first release includes:\n\n`isError` field.\nHere's another example.\n\nSuppose a tool is expected to return an MCP tool error, but instead reports success.\n\nMCP Verdict can detect that mismatch:\n\n```\nknown-failure: FAIL: Expected tool_error, observed success\nSummary: 0 passed, 1 failed\n```\n\nThis is the behavior I find most interesting: **testing not just whether an MCP tool works, but whether it fails correctly.**\n\nMCP Verdict v0.1.0 is now available on PyPI.\n\nRequires Python 3.11 or 3.12.\n\n```\npip install mcp-verdict\n```\n\nVerify the installation:\n\n```\nmcp-verdict --help\n```\n\nFor runnable examples, clone the repository:\n\n```\ngit clone https://github.com/kuruash/mcp-verdict.git\ncd mcp-verdict\nmcp-verdict check --config examples/basic_stdio/verdict.yaml\n```\n\nThe repository also includes calculator, intentional error-handling, and local HTTP security examples.\n\nBuilding this made me appreciate how much work goes into handling the unhappy paths of software.\n\nConnecting to a server is one thing. Handling timeouts, unexpected disconnects, malformed responses, authentication failures, and cleanup reliably is a different challenge.\n\nI also learned a lot about Python packaging, GitHub Actions, testing with real subprocesses, and publishing through PyPI Trusted Publishing.\n\nThere's still plenty to improve. MCP Verdict isn't a comprehensive MCP security scanner, and it doesn't attempt to audit arbitrary OAuth/JWT implementations.\n\nFor now, the goal is simple: make failure scenarios reproducible and help developers catch regressions.\n\nIf you're building or maintaining MCP servers, I'd love to know:\n\n**What's the most frustrating failure scenario you've encountered while testing an MCP server?**\n\nI'm especially interested in edge cases that MCP Verdict doesn't handle yet.\n\nGitHub: [https://github.com/kuruash/mcp-verdict](https://github.com/kuruash/mcp-verdict)\n\nPyPI: [https://pypi.org/project/mcp-verdict/](https://pypi.org/project/mcp-verdict/)\n\nIssues, feedback, and contributions are welcome!", "url": "https://wpnews.pro/news/i-built-an-open-source-tool-to-test-what-happens-when-mcp-servers-fail", "canonical_source": "https://dev.to/kuruash/i-built-an-open-source-tool-to-test-what-happens-when-mcp-servers-fail-g1g", "published_at": "2026-11-11 05:39:14+00:00", "updated_at": "2026-11-11 05:51:11.068748+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "developer-tools", "ai-tools", "mlops"], "entities": ["MCP Verdict", "Model Context Protocol", "PyPI", "GitHub", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-an-open-source-tool-to-test-what-happens-when-mcp-servers-fail", "markdown": "https://wpnews.pro/news/i-built-an-open-source-tool-to-test-what-happens-when-mcp-servers-fail.md", "text": "https://wpnews.pro/news/i-built-an-open-source-tool-to-test-what-happens-when-mcp-servers-fail.txt", "jsonld": "https://wpnews.pro/news/i-built-an-open-source-tool-to-test-what-happens-when-mcp-servers-fail.jsonld"}}