{"slug": "show-hn-mcp-crashtest-multi-step-stateful-contract-fuzzer-for-mcp-servers", "title": "Show HN: MCP-CrashTest – Multi-step stateful contract fuzzer for MCP servers", "summary": "A developer released MCP-CrashTest v0.2.1, an open-source stateful contract fuzzer that runs guided multi-step random walks against Model Context Protocol servers inside ephemeral OS sandboxes and outputs deduplicated minimal JSON-RPC reproducers. Testing on 2026-11-20 in both `--adapter none` zero-shot mode and `--adapter auto` found that `mcp-server-sqlite` returns `isError: false` on SQL failures such as UNIQUE constraint violations, and that `mcp-server-git` passes human CLI formatting like \"* main\" from `git_branch` into `git_checkout`, `git_diff`, `git_show` and `git_merge`, which then fail. The tool requires Python >= 3.10 and targets agents that chain 5–20 tool calls, where single-step unit tests miss cross-tool contract breaks and stdio deadlocks.", "body_md": "**Unit tests check single tool calls. Real agents fail on multi-step contracts.**\n\n*State-aware random walks with POSIX inode-preserving rollback for Model Context Protocol (MCP) servers.*\n\nTools like MCP Inspector and standard unit tests validate **single-step static calls**. In production, AI agents execute **5–20 step tool chains** where outputs from discovery/list tools feed downstream mutations, entities collide, and `stdio` pipes hang.\n\nInspired by reversing the state-aware random walk paradigm from **ToolSpace** ([arXiv:2611.05036](https://arxiv.org/abs/2611.05036))—instead of keeping successful trajectories for model training and discarding rollbacks—`MCP-CrashTest` mines a dependency graph from `tools/list`, runs guided multi-step random walks inside ephemeral OS sandboxes, rolls back on failure to explore neighboring branches, prunes noisy intermediate steps, and outputs deduplicated **minimal JSON-RPC reproducers**.\n\n1. \n**Masked Errors (`isError: false` on SQL/Runtime Failures) — Protocol Violation (`CRITICAL`)**\n In`mcp-server-sqlite` (verified on**2026-11-20** with`MCP-CrashTest v0.2.1` in**both**`--adapter none` pure zero-shot mode and`--adapter auto` ), SQL errors return`isError: false` , causing agents to treat error strings as valid query results (context poisoning):\n\n```\n{\n  \"method\": \"tools/call\",\n  \"params\": {\n    \"name\": \"write_query\",\n    \"arguments\": {\n      \"query\": \"INSERT INTO users (id, name, email) VALUES (1, 'CollisionBob', 'bob_new@example.com');\"\n    }\n  },\n  \"response\": {\n    \"text\": \"Database error: UNIQUE constraint failed: users.id\",\n    \"isError\": false\n  }\n}\n```\n\n2. \n**CLI Output Leaks Across Tools — Cross-Tool Contract Break (`HIGH`)**\n In`mcp-server-git` ,`git_branch` outputs human CLI formatting (`\"* main\"` ). Passing that branch name directly to`git_checkout` ,`git_diff` ,`git_show` , or`git_merge` fails:\n\n```\n[\n  { \"tool\": \"git_branch\", \"response\": { \"text\": \"* main\", \"isError\": false } },\n  { \"tool\": \"git_merge\", \"arguments\": { \"source_branch\": \"* main\" }, \"response\": { \"text\": \"Error executing tool git_merge: Ref '* main' did not resolve to an object\", \"isError\": true } }\n]\n```\n\n3. \n**Silent State Corruption & Stdio Deadlocks — Protocol Violations (`CRITICAL`)**\n Read-after-write verification catches tools that report`isError: false` without persisting state, as well as hung`stdio` buffers or severed child-process pipes (with automatic subprocess recycling).\n4. \n**Agent Contract Smells — UX & Consistency Pitfalls (`MEDIUM` / `LOW` / `INFO`)**\n Explicitly separated from protocol bugs: undocumented glob-only`search_files` matching, relative vs. parent-joined path contracts, no-upsert skip behavior (`create_entities` ), and intra-server`isError` inconsistencies.\n\nRequires **Python >= 3.10**.\n\n```\n# Run directly without cloning (once published on PyPI)\nuvx mcp-crashtest --target sqlite --walks 15 --depth 6\n\n# Or install from source\ngit clone https://github.com/SnowyLumiere/mcp-crashtest.git\ncd mcp-crashtest\npip install -e \".[dev]\"\n```\n\nUse `--` to pass the server command cleanly without option collisions, and `--adapter none` to run **pure zero-shot generic stateful fuzzing** with zero domain-specific assumptions:\n\n```\nmcp-crashtest \\\n  --target my-server \\\n  --adapter none \\\n  --sandbox directory \\\n  --seed 42 \\\n  --walks 20 \\\n  --depth 6 \\\n  --fail-on-crash \\\n  --output reports/my_server_report.md \\\n  -- python3 -m my_mcp_server --root {sandbox}\n```\n\nFor stateless/API-only servers, use `--sandbox none` (context-memory rollback and branch exploration still run):\n\n```\nmcp-crashtest \\\n  --target my-api-server \\\n  --adapter none \\\n  --sandbox none \\\n  --walks 20 \\\n  -- node dist/index.js --env test\n```\n\nRequires `npx` (`filesystem` / `memory`) or `uvx` + `git` (`sqlite` / `git`):\n\n```\nmcp-crashtest --target sqlite --walks 15 --depth 6 --output reports/sqlite_crash_report.md\nmcp-crashtest --target git --walks 15 --depth 6 --output reports/git_crash_report.md\nmcp-crashtest --target filesystem --walks 15 --depth 6 --output reports/filesystem_crash_report.md\nmcp-crashtest --target memory --walks 15 --depth 6 --output reports/memory_crash_report.md\n```\n\n`MCP-CrashTest` separates the **generic zero-shot stateful engine** from optional **domain contract adapters** (`mcp_crashtest/adapters/`):\n\n| Capability | Generic Mode ( `--adapter none` ) | Built-in Domain Adapters ( `--adapter auto` ) | \n|---|---|---|\n| **Schema Graph Mining** | Infers `CREATE` /`READ` /`MUTATE` /`DELETE` +`DATA` /`STATE` /`RETRY` edges from`tools/list` ; handles Pydantic v2`anyOf` /`oneOf` /`enum` and`properties: null` | Adds domain-specific edge bindings for `filesystem` ,`sqlite` ,`memory` ,`git` | \n| **Context Flow & Fuzzing** | Extracts IDs/keys/URIs/paths from args & JSON responses; injects boundary strings, traversal paths, extreme ints | Adds domain-specific probes (e.g. PK collisions, glob-vs-substring queries, raw `* main` branch passing) | \n| **Post-Condition Probes** | Probe A ( `RETRY_TRAP` ), Generic Probe B (Read-After-Write on`id` /`key` /`uri` ), Probe C (`SCHEMA_BREAK` ), Probe D (`STDIO_DEADLOCK` ) | Adds calibrated `CONTRACT_SMELL` checks (e.g.`search_files` glob pitfall,`describe_table``[]` ,`create_entities` no-upsert skip) | \n| **Report Provenance** | Every finding is tagged with `source: generic` | Every adapter finding is explicitly tagged `source: adapter:<name>` | \n\n| `--sandbox` | CLI Placeholders | Injected Env Var | Seeded Initial Data | \n|---|---|---|---|\n| `directory`*(default)* | `{sandbox}` ,`{work_dir}` | `MCP_SANDBOX_DIR` | `sample.txt` ,`data_dir/config.json` ,`data_dir/nested_note.txt` | \n| `git` | `{sandbox}` ,`{work_dir}` | `MCP_SANDBOX_DIR` | Initialized `main` branch + initial commit (`commit.gpgsign=false` ) | \n| `sqlite` | `{sandbox}` ,`{db_path}` | `MCP_SANDBOX_DB_PATH` | `test_target.db` with seeded`users` table (2 rows) | \n| `memory` | `{sandbox}` ,`{memory_path}` | `MEMORY_FILE_PATH` | Isolated temp path `memory_graph.json` | \n| `none` | — | — | No local filesystem sandbox (context rollback still active) | \n\nVerified on **2026-11-20** (`--walks 15 --depth 6 --rollbacks 3 --seed 42`, Python 3.12, `mcp` SDK v2.3.0 client with full v1.x/v2.x compatibility; pruned reports in [reports/](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports)):\n\n| Server | Mode | Protocol Violations | Contract Smells | Headline Minimal Reproducers | Report | \n|---|---|---|---|---|---|\n| **`mcp-server-sqlite`** | `--adapter none`*(pure zero-shot)* | **` 3`** (` CRITICAL` ) | `0` | `write_query` ,`create_table` ,`read_query` return`\"Error: ...\"` with`isError: false` (1-step each,`source: generic` ) | — | \n| **`mcp-server-sqlite`** | `--adapter auto` | **`3`** (` CRITICAL` ) | `1` (`MEDIUM` ) | SQL errors return `isError: false` (×3, 1-step);`describe_table` returns`[]` on missing table (1-step) | [sqlite](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/sqlite_crash_report.md) | \n| **`mcp-server-git`** | `--adapter auto` | `0` | **`4`** (` HIGH``SCHEMA_BREAK` ) | `git_branch` leaks`'* main'` , breaking`git_checkout` ,`git_diff` ,`git_show` ,`git_merge` (2-step each) | [git](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/git_crash_report.md) | \n| **`@modelcontextprotocol/server-filesystem`** | `--adapter auto` | `0` | `4` (`MEDIUM` ) | `list_directory` bare filenames vs`read_file` /`read_text_file` /`get_file_info` (2-step);`search_files` glob-only miss (1-step) | [filesystem](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/filesystem_crash_report.md) | \n| **`@modelcontextprotocol/server-memory`** | `--adapter auto` | `0` | `3` (`LOW` /`INFO` ) | `create_entities` skips existing entities with`isError: false` (2-step);`delete_*` idempotent`0` differs from`add_*` (1-step) | [memory](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/memory_crash_report.md) | \n\n**Notes**:\n\n- The\n`sqlite` preset launches `uvx --with \"mcp<2\" mcp-server-sqlite` because upstream `mcp-server-sqlite` currently uses the `mcp 1.x` `@server.list_resources()` API and fails to boot under `mcp 2.x`. `MCP-CrashTest` itself supports both `mcp` SDK v1.x (`inputSchema`/` isError`) and v2.x (` input_schema`/` is_error`).\n[reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md](https://github.com/snowyLumiere/mcp-crashtest/blob/main/reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md) is preserved as the historical v0.2.0 archive.\n\n1. **Strict `isError` Semantics** : Never return`isError: false` on SQL errors, constraint violations, or failed mutations.\n2. **Cross-Tool I/O Symmetry** : Discovery/list tool outputs must be directly consumable by downstream tools without CLI decorations (`* main` ) or stripped parent paths.\n3. **Explicit Matching & Upsert Contracts** : Document glob vs. substring matching and provide explicit upsert/conflict errors instead of ambiguous`isError: false` skips.\n4. **Actionable Errors & Pipe Backpressure** : Return offending parameter names and suggested corrective tools; paginate large responses to prevent`stdio` deadlocks.\n\n| Flag | Default | Description | \n|---|---|---|\n| `--target` | `\"custom\"` | Target label or built-in preset ( `filesystem` ,`sqlite` ,`memory` ,`git` ) | \n| `--adapter` | `\"auto\"` | Domain adapter mode: `auto` ,`none` (pure generic), or explicit adapter name(s) | \n| `--cmd` /`-- <cmd...>` | `None` | Server executable (prefer POSIX `--` passthrough to avoid flag clashes) | \n| `--sandbox` | `\"directory\"` | `directory` \\|`sqlite` \\|`git` \\|`memory` \\|`none` | \n| `--walks` | `15` | Number of random walk chains | \n| `--depth` | `6` | Max step depth per walk | \n| `--rollbacks` | `3` | Max rollback retries per walk | \n| `--fuzz-ratio` | `0.25` | Boundary fuzzing probability ( `0.0` –`1.0` ) | \n| `--timeout` | `8.0` | Per-step & `tools/list` timeout in seconds | \n| `--init-timeout` | `20.0` | Session `initialize()` timeout in seconds | \n| `--seed` | `None` | Local deterministic RNG seed (does not mutate global `random` state) | \n| `--lang` | `\"en\"` | Report language: `en` \\|`zh` | \n| `--fail-on-crash` | `False` | Exit with code `1` if findings matching`--fail-on` are detected (for CI gates) | \n| `--fail-on` | `\"any\"` | CI failure scope: `any` (all findings) \\|`protocol` (`PROTOCOL_VIOLATION` only) | \n| `--output` | `\"reports/mcp_crash_report.md\"` | Output Markdown report path | \n| `--version` | — | Print version and exit | \n\n- **Transport** : Currently supports**`stdio`** transport only.**Streamable HTTP / SSE** support is on the roadmap.\n- **SQLite Sandbox** : In-place binary file rollback assumes per-call SQLite connections (such as`mcp-server-sqlite` ). Long-lived connection pools with in-memory page caches require session restarts between walks.\n- **Process Memory** : For servers keeping unpersisted in-process state, sandbox rollback restores disk state only.\n\n``` python\nimport asyncio\nfrom mcp_crashtest.cli import run_crash_test\n\nasync def main():\n    crashes = await run_crash_test(\n        cmd=\"python3\",\n        args=[\"my_server.py\", \"--root\", \"{sandbox}\"],\n        target_name=\"my-server\",\n        adapter_mode=\"none\",\n        sandbox_type=\"directory\",\n        seed=42,\n        lang=\"en\",\n        output_report_path=\"reports/my_server.md\",\n    )\n    print(f\"Hits: {len(crashes)} | Unique: {len({c.fingerprint for c in crashes})}\")\n\nasyncio.run(main())\n```\n\nRun the test suite (zero third-party dependencies required for `unittest`):\n\n```\nPYTHONPATH=. python3 -m unittest discover -s tests -v\n# or with pytest\nPYTHONPATH=. pytest tests/ -v\n```\n\nReleased under the [MIT License](https://github.com/snowyLumiere/mcp-crashtest/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-mcp-crashtest-multi-step-stateful-contract-fuzzer-for-mcp-servers", "canonical_source": "https://github.com/SnowyLumiere/mcp-crashtest", "published_at": "2026-11-20 07:55:20+00:00", "updated_at": "2026-11-20 08:14:37.256684+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools", "ai-safety"], "entities": ["MCP-CrashTest", "Model Context Protocol", "mcp-server-sqlite", "mcp-server-git", "MCP Inspector", "ToolSpace", "arXiv:2611.05036", "Python"], "also_reported_by": [], "takeaway": "MCP-CrashTest v0.2.1 fuzzes 5–20 step MCP tool chains with state-aware random walks and POSIX inode-preserving rollback, and its first runs flagged `mcp-server-sqlite` returning `isError: false` on SQL errors and `mcp-server-git` leaking \"* main\" CLI formatting into downstream tools.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/show-hn-mcp-crashtest-multi-step-stateful-contract-fuzzer-for-mcp-servers", "markdown": "https://wpnews.pro/news/show-hn-mcp-crashtest-multi-step-stateful-contract-fuzzer-for-mcp-servers.md", "text": "https://wpnews.pro/news/show-hn-mcp-crashtest-multi-step-stateful-contract-fuzzer-for-mcp-servers.txt", "jsonld": "https://wpnews.pro/news/show-hn-mcp-crashtest-multi-step-stateful-contract-fuzzer-for-mcp-servers.jsonld"}}