I Built an Open-Source Tool to Test What Happens When MCP Servers Fail

A developer released MCP Verdict, an open-source Python CLI on PyPI that tests whether Model Context Protocol servers report failures correctly. Developers define expected server behavior in YAML — such as whether a tool should return an error or success — and the tool connects to the server, runs the tool, and flags mismatches, like a tool returning "Operation failed" while setting isError to false. Version 0.1.0 requires Python 3.11 or 3.12 and ships with calculator, error-handling, and local HTTP security examples.

I've been spending a lot of time exploring MCP servers and AI agents lately. One thing that caught my attention: we spend a lot of time testing whether tools work, but what about when they don't? What happens when an MCP tool crashes? When it receives invalid arguments? When a protected endpoint receives an unauthorized request? And more importantly, does the server actually report those failures correctly? That's what led me to build MCP Verdict . Imagine an MCP tool fails internally but returns something like this: { "isError": false, "content": { "type": "text", "text": "Operation failed" } } The message says the operation failed, but the MCP response reports success. That inconsistency can cause problems for clients and agents relying on the tool's result. And it's exactly the kind of behavior that's worth testing before deploying an MCP server. MCP Verdict https://github.com/kuruash/mcp-verdict is an open-source Python CLI that lets developers define expected MCP server behavior and check what actually happens. Instead of writing separate scripts for every failure scenario, you describe your expectations in YAML. For example: server: command: - python - examples/basic stdio/server.py tool: name: ping arguments: {} expected: success Then run: mcp-verdict check --config verdict.yaml MCP Verdict connects to the server, executes the tool, and checks whether the actual result matches your expectation. The first release includes: isError field. Here's another example. Suppose a tool is expected to return an MCP tool error, but instead reports success. MCP Verdict can detect that mismatch: known-failure: FAIL: Expected tool error, observed success Summary: 0 passed, 1 failed This is the behavior I find most interesting: testing not just whether an MCP tool works, but whether it fails correctly. MCP Verdict v0.1.0 is now available on PyPI. Requires Python 3.11 or 3.12. pip install mcp-verdict Verify the installation: mcp-verdict --help For runnable examples, clone the repository: git clone https://github.com/kuruash/mcp-verdict.git cd mcp-verdict mcp-verdict check --config examples/basic stdio/verdict.yaml The repository also includes calculator, intentional error-handling, and local HTTP security examples. Building this made me appreciate how much work goes into handling the unhappy paths of software. Connecting to a server is one thing. Handling timeouts, unexpected disconnects, malformed responses, authentication failures, and cleanup reliably is a different challenge. I also learned a lot about Python packaging, GitHub Actions, testing with real subprocesses, and publishing through PyPI Trusted Publishing. There's still plenty to improve. MCP Verdict isn't a comprehensive MCP security scanner, and it doesn't attempt to audit arbitrary OAuth/JWT implementations. For now, the goal is simple: make failure scenarios reproducible and help developers catch regressions. If you're building or maintaining MCP servers, I'd love to know: What's the most frustrating failure scenario you've encountered while testing an MCP server? I'm especially interested in edge cases that MCP Verdict doesn't handle yet. GitHub: https://github.com/kuruash/mcp-verdict https://github.com/kuruash/mcp-verdict PyPI: https://pypi.org/project/mcp-verdict/ https://pypi.org/project/mcp-verdict/ Issues, feedback, and contributions are welcome