# I Built an Open-Source Tool to Test What Happens When MCP Servers Fail

> Source: <https://dev.to/kuruash/i-built-an-open-source-tool-to-test-what-happens-when-mcp-servers-fail-g1g>
> Published: 2026-11-11 05:39:14+00:00

I've been spending a lot of time exploring MCP servers and AI agents lately.

One thing that caught my attention: **we spend a lot of time testing whether tools work, but what about when they don't?**

What happens when an MCP tool crashes? When it receives invalid arguments? When a protected endpoint receives an unauthorized request?

And more importantly, does the server actually report those failures correctly?

That's what led me to build **MCP Verdict**.

Imagine an MCP tool fails internally but returns something like this:

```
{
  "isError": false,
  "content": [
    {
      "type": "text",
      "text": "Operation failed"
    }
  ]
}
```

The message says the operation failed, but the MCP response reports success.

That inconsistency can cause problems for clients and agents relying on the tool's result.

And it's exactly the kind of behavior that's worth testing before deploying an MCP server.

[MCP Verdict](https://github.com/kuruash/mcp-verdict) is an open-source Python CLI that lets developers define expected MCP server behavior and check what actually happens.

Instead of writing separate scripts for every failure scenario, you describe your expectations in YAML.

For example:

```
server:
  command:
    - python
    - examples/basic_stdio/server.py

tool:
  name: ping
  arguments: {}

expected: success
```

Then run:

```
mcp-verdict check --config verdict.yaml
```

MCP Verdict connects to the server, executes the tool, and checks whether the actual result matches your expectation.

The first release includes:

`isError` field.
Here's another example.

Suppose a tool is expected to return an MCP tool error, but instead reports success.

MCP Verdict can detect that mismatch:

```
known-failure: FAIL: Expected tool_error, observed success
Summary: 0 passed, 1 failed
```

This is the behavior I find most interesting: **testing not just whether an MCP tool works, but whether it fails correctly.**

MCP Verdict v0.1.0 is now available on PyPI.

Requires Python 3.11 or 3.12.

```
pip install mcp-verdict
```

Verify the installation:

```
mcp-verdict --help
```

For runnable examples, clone the repository:

```
git clone https://github.com/kuruash/mcp-verdict.git
cd mcp-verdict
mcp-verdict check --config examples/basic_stdio/verdict.yaml
```

The repository also includes calculator, intentional error-handling, and local HTTP security examples.

Building this made me appreciate how much work goes into handling the unhappy paths of software.

Connecting to a server is one thing. Handling timeouts, unexpected disconnects, malformed responses, authentication failures, and cleanup reliably is a different challenge.

I also learned a lot about Python packaging, GitHub Actions, testing with real subprocesses, and publishing through PyPI Trusted Publishing.

There's still plenty to improve. MCP Verdict isn't a comprehensive MCP security scanner, and it doesn't attempt to audit arbitrary OAuth/JWT implementations.

For now, the goal is simple: make failure scenarios reproducible and help developers catch regressions.

If you're building or maintaining MCP servers, I'd love to know:

**What's the most frustrating failure scenario you've encountered while testing an MCP server?**

I'm especially interested in edge cases that MCP Verdict doesn't handle yet.

GitHub: [https://github.com/kuruash/mcp-verdict](https://github.com/kuruash/mcp-verdict)

PyPI: [https://pypi.org/project/mcp-verdict/](https://pypi.org/project/mcp-verdict/)

Issues, feedback, and contributions are welcome!