# Show HN: MCP-CrashTest – Multi-step stateful contract fuzzer for MCP servers

> Source: <https://github.com/SnowyLumiere/mcp-crashtest>
> Published: 2026-11-20 07:55:20+00:00

**Key takeaway:** MCP-CrashTest v0.2.1 fuzzes 5–20 step MCP tool chains with state-aware random walks and POSIX inode-preserving rollback, and its first runs flagged `mcp-server-sqlite` returning `isError: false` on SQL errors and `mcp-server-git` leaking "* main" CLI formatting into downstream tools.

**Unit tests check single tool calls. Real agents fail on multi-step contracts.**

*State-aware random walks with POSIX inode-preserving rollback for Model Context Protocol (MCP) servers.*

Tools like MCP Inspector and standard unit tests validate **single-step static calls**. In production, AI agents execute **5–20 step tool chains** where outputs from discovery/list tools feed downstream mutations, entities collide, and `stdio` pipes hang.

Inspired by reversing the state-aware random walk paradigm from **ToolSpace** ([arXiv:2611.05036](https://arxiv.org/abs/2611.05036))—instead of keeping successful trajectories for model training and discarding rollbacks—`MCP-CrashTest` mines a dependency graph from `tools/list`, runs guided multi-step random walks inside ephemeral OS sandboxes, rolls back on failure to explore neighboring branches, prunes noisy intermediate steps, and outputs deduplicated **minimal JSON-RPC reproducers**.

1. 
**Masked Errors (`isError: false` on SQL/Runtime Failures) — Protocol Violation (`CRITICAL`)**
 In`mcp-server-sqlite` (verified on**2026-11-20** with`MCP-CrashTest v0.2.1` in**both**`--adapter none` pure zero-shot mode and`--adapter auto` ), SQL errors return`isError: false` , causing agents to treat error strings as valid query results (context poisoning):

```
{
  "method": "tools/call",
  "params": {
    "name": "write_query",
    "arguments": {
      "query": "INSERT INTO users (id, name, email) VALUES (1, 'CollisionBob', 'bob_new@example.com');"
    }
  },
  "response": {
    "text": "Database error: UNIQUE constraint failed: users.id",
    "isError": false
  }
}
```

2. 
**CLI Output Leaks Across Tools — Cross-Tool Contract Break (`HIGH`)**
 In`mcp-server-git` ,`git_branch` outputs human CLI formatting (`"* main"` ). Passing that branch name directly to`git_checkout` ,`git_diff` ,`git_show` , or`git_merge` fails:

```
[
  { "tool": "git_branch", "response": { "text": "* main", "isError": false } },
  { "tool": "git_merge", "arguments": { "source_branch": "* main" }, "response": { "text": "Error executing tool git_merge: Ref '* main' did not resolve to an object", "isError": true } }
]
```

3. 
**Silent State Corruption & Stdio Deadlocks — Protocol Violations (`CRITICAL`)**
 Read-after-write verification catches tools that report`isError: false` without persisting state, as well as hung`stdio` buffers or severed child-process pipes (with automatic subprocess recycling).
4. 
**Agent Contract Smells — UX & Consistency Pitfalls (`MEDIUM` / `LOW` / `INFO`)**
 Explicitly separated from protocol bugs: undocumented glob-only`search_files` matching, relative vs. parent-joined path contracts, no-upsert skip behavior (`create_entities` ), and intra-server`isError` inconsistencies.

Requires **Python >= 3.10**.

```
# Run directly without cloning (once published on PyPI)
uvx mcp-crashtest --target sqlite --walks 15 --depth 6

# Or install from source
git clone https://github.com/SnowyLumiere/mcp-crashtest.git
cd mcp-crashtest
pip install -e ".[dev]"
```

Use `--` to pass the server command cleanly without option collisions, and `--adapter none` to run **pure zero-shot generic stateful fuzzing** with zero domain-specific assumptions:

```
mcp-crashtest \
  --target my-server \
  --adapter none \
  --sandbox directory \
  --seed 42 \
  --walks 20 \
  --depth 6 \
  --fail-on-crash \
  --output reports/my_server_report.md \
  -- python3 -m my_mcp_server --root {sandbox}
```

For stateless/API-only servers, use `--sandbox none` (context-memory rollback and branch exploration still run):

```
mcp-crashtest \
  --target my-api-server \
  --adapter none \
  --sandbox none \
  --walks 20 \
  -- node dist/index.js --env test
```

Requires `npx` (`filesystem` / `memory`) or `uvx` + `git` (`sqlite` / `git`):

```
mcp-crashtest --target sqlite --walks 15 --depth 6 --output reports/sqlite_crash_report.md
mcp-crashtest --target git --walks 15 --depth 6 --output reports/git_crash_report.md
mcp-crashtest --target filesystem --walks 15 --depth 6 --output reports/filesystem_crash_report.md
mcp-crashtest --target memory --walks 15 --depth 6 --output reports/memory_crash_report.md
```

`MCP-CrashTest` separates the **generic zero-shot stateful engine** from optional **domain contract adapters** (`mcp_crashtest/adapters/`):

| Capability | Generic Mode ( `--adapter none` ) | Built-in Domain Adapters ( `--adapter auto` ) | 
|---|---|---|
| **Schema Graph Mining** | Infers `CREATE` /`READ` /`MUTATE` /`DELETE` +`DATA` /`STATE` /`RETRY` edges from`tools/list` ; handles Pydantic v2`anyOf` /`oneOf` /`enum` and`properties: null` | Adds domain-specific edge bindings for `filesystem` ,`sqlite` ,`memory` ,`git` | 
| **Context Flow & Fuzzing** | Extracts IDs/keys/URIs/paths from args & JSON responses; injects boundary strings, traversal paths, extreme ints | Adds domain-specific probes (e.g. PK collisions, glob-vs-substring queries, raw `* main` branch passing) | 
| **Post-Condition Probes** | Probe A ( `RETRY_TRAP` ), Generic Probe B (Read-After-Write on`id` /`key` /`uri` ), Probe C (`SCHEMA_BREAK` ), Probe D (`STDIO_DEADLOCK` ) | Adds calibrated `CONTRACT_SMELL` checks (e.g.`search_files` glob pitfall,`describe_table``[]` ,`create_entities` no-upsert skip) | 
| **Report Provenance** | Every finding is tagged with `source: generic` | Every adapter finding is explicitly tagged `source: adapter:<name>` | 

| `--sandbox` | CLI Placeholders | Injected Env Var | Seeded Initial Data | 
|---|---|---|---|
| `directory`*(default)* | `{sandbox}` ,`{work_dir}` | `MCP_SANDBOX_DIR` | `sample.txt` ,`data_dir/config.json` ,`data_dir/nested_note.txt` | 
| `git` | `{sandbox}` ,`{work_dir}` | `MCP_SANDBOX_DIR` | Initialized `main` branch + initial commit (`commit.gpgsign=false` ) | 
| `sqlite` | `{sandbox}` ,`{db_path}` | `MCP_SANDBOX_DB_PATH` | `test_target.db` with seeded`users` table (2 rows) | 
| `memory` | `{sandbox}` ,`{memory_path}` | `MEMORY_FILE_PATH` | Isolated temp path `memory_graph.json` | 
| `none` | — | — | No local filesystem sandbox (context rollback still active) | 

Verified on **2026-11-20** (`--walks 15 --depth 6 --rollbacks 3 --seed 42`, Python 3.12, `mcp` SDK v2.3.0 client with full v1.x/v2.x compatibility; pruned reports in [reports/](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports)):

| Server | Mode | Protocol Violations | Contract Smells | Headline Minimal Reproducers | Report | 
|---|---|---|---|---|---|
| **`mcp-server-sqlite`** | `--adapter none`*(pure zero-shot)* | **` 3`** (` CRITICAL` ) | `0` | `write_query` ,`create_table` ,`read_query` return`"Error: ..."` with`isError: false` (1-step each,`source: generic` ) | — | 
| **`mcp-server-sqlite`** | `--adapter auto` | **`3`** (` CRITICAL` ) | `1` (`MEDIUM` ) | SQL errors return `isError: false` (×3, 1-step);`describe_table` returns`[]` on missing table (1-step) | [sqlite](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/sqlite_crash_report.md) | 
| **`mcp-server-git`** | `--adapter auto` | `0` | **`4`** (` HIGH``SCHEMA_BREAK` ) | `git_branch` leaks`'* main'` , breaking`git_checkout` ,`git_diff` ,`git_show` ,`git_merge` (2-step each) | [git](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/git_crash_report.md) | 
| **`@modelcontextprotocol/server-filesystem`** | `--adapter auto` | `0` | `4` (`MEDIUM` ) | `list_directory` bare filenames vs`read_file` /`read_text_file` /`get_file_info` (2-step);`search_files` glob-only miss (1-step) | [filesystem](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/filesystem_crash_report.md) | 
| **`@modelcontextprotocol/server-memory`** | `--adapter auto` | `0` | `3` (`LOW` /`INFO` ) | `create_entities` skips existing entities with`isError: false` (2-step);`delete_*` idempotent`0` differs from`add_*` (1-step) | [memory](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/memory_crash_report.md) | 

**Notes**:

- The
`sqlite` preset launches `uvx --with "mcp<2" mcp-server-sqlite` because upstream `mcp-server-sqlite` currently uses the `mcp 1.x` `@server.list_resources()` API and fails to boot under `mcp 2.x`. `MCP-CrashTest` itself supports both `mcp` SDK v1.x (`inputSchema`/` isError`) and v2.x (` input_schema`/` is_error`).
[reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md) is preserved as the historical v0.2.0 archive.

1. **Strict `isError` Semantics** : Never return`isError: false` on SQL errors, constraint violations, or failed mutations.
2. **Cross-Tool I/O Symmetry** : Discovery/list tool outputs must be directly consumable by downstream tools without CLI decorations (`* main` ) or stripped parent paths.
3. **Explicit Matching & Upsert Contracts** : Document glob vs. substring matching and provide explicit upsert/conflict errors instead of ambiguous`isError: false` skips.
4. **Actionable Errors & Pipe Backpressure** : Return offending parameter names and suggested corrective tools; paginate large responses to prevent`stdio` deadlocks.

| Flag | Default | Description | 
|---|---|---|
| `--target` | `"custom"` | Target label or built-in preset ( `filesystem` ,`sqlite` ,`memory` ,`git` ) | 
| `--adapter` | `"auto"` | Domain adapter mode: `auto` ,`none` (pure generic), or explicit adapter name(s) | 
| `--cmd` /`-- <cmd...>` | `None` | Server executable (prefer POSIX `--` passthrough to avoid flag clashes) | 
| `--sandbox` | `"directory"` | `directory` \|`sqlite` \|`git` \|`memory` \|`none` | 
| `--walks` | `15` | Number of random walk chains | 
| `--depth` | `6` | Max step depth per walk | 
| `--rollbacks` | `3` | Max rollback retries per walk | 
| `--fuzz-ratio` | `0.25` | Boundary fuzzing probability ( `0.0` –`1.0` ) | 
| `--timeout` | `8.0` | Per-step & `tools/list` timeout in seconds | 
| `--init-timeout` | `20.0` | Session `initialize()` timeout in seconds | 
| `--seed` | `None` | Local deterministic RNG seed (does not mutate global `random` state) | 
| `--lang` | `"en"` | Report language: `en` \|`zh` | 
| `--fail-on-crash` | `False` | Exit with code `1` if findings matching`--fail-on` are detected (for CI gates) | 
| `--fail-on` | `"any"` | CI failure scope: `any` (all findings) \|`protocol` (`PROTOCOL_VIOLATION` only) | 
| `--output` | `"reports/mcp_crash_report.md"` | Output Markdown report path | 
| `--version` | — | Print version and exit | 

- **Transport** : Currently supports**`stdio`** transport only.**Streamable HTTP / SSE** support is on the roadmap.
- **SQLite Sandbox** : In-place binary file rollback assumes per-call SQLite connections (such as`mcp-server-sqlite` ). Long-lived connection pools with in-memory page caches require session restarts between walks.
- **Process Memory** : For servers keeping unpersisted in-process state, sandbox rollback restores disk state only.

``` python
import asyncio
from mcp_crashtest.cli import run_crash_test

async def main():
    crashes = await run_crash_test(
        cmd="python3",
        args=["my_server.py", "--root", "{sandbox}"],
        target_name="my-server",
        adapter_mode="none",
        sandbox_type="directory",
        seed=42,
        lang="en",
        output_report_path="reports/my_server.md",
    )
    print(f"Hits: {len(crashes)} | Unique: {len({c.fingerprint for c in crashes})}")

asyncio.run(main())
```

Run the test suite (zero third-party dependencies required for `unittest`):

```
PYTHONPATH=. python3 -m unittest discover -s tests -v
# or with pytest
PYTHONPATH=. pytest tests/ -v
```

Released under the [MIT License](https://github.com/snowyLumiere/mcp-crashtest/blob/main/LICENSE).
