Unit tests check single tool calls. Real agents fail on multi-step contracts.
State-aware random walks with POSIX inode-preserving rollback for Model Context Protocol (MCP) servers.
Tools like MCP Inspector and standard unit tests validate single-step static calls. In production, AI agents execute 5–20 step tool chains where outputs from discovery/list tools feed downstream mutations, entities collide, and stdio pipes hang.
Inspired by reversing the state-aware random walk paradigm from ToolSpace (arXiv:2611.05036)—instead of keeping successful trajectories for model training and discarding rollbacks—MCP-CrashTest mines a dependency graph from tools/list, runs guided multi-step random walks inside ephemeral OS sandboxes, rolls back on failure to explore neighboring branches, prunes noisy intermediate steps, and outputs deduplicated minimal JSON-RPC reproducers.
Masked Errors (isError: false on SQL/Runtime Failures) — Protocol Violation (CRITICAL)
Inmcp-server-sqlite (verified on2026-11-20 withMCP-CrashTest v0.2.1 inboth--adapter none pure zero-shot mode and--adapter auto ), SQL errors returnisError: false , causing agents to treat error strings as valid query results (context poisoning):
{
"method": "tools/call",
"params": {
"name": "write_query",
"arguments": {
"query": "INSERT INTO users (id, name, email) VALUES (1, 'CollisionBob', 'bob_new@example.com');"
}
},
"response": {
"text": "Database error: UNIQUE constraint failed: users.id",
"isError": false
}
}
CLI Output Leaks Across Tools — Cross-Tool Contract Break (HIGH)
Inmcp-server-git ,git_branch outputs human CLI formatting ("* main" ). Passing that branch name directly togit_checkout ,git_diff ,git_show , orgit_merge fails:
[
{ "tool": "git_branch", "response": { "text": "* main", "isError": false } },
{ "tool": "git_merge", "arguments": { "source_branch": "* main" }, "response": { "text": "Error executing tool git_merge: Ref '* main' did not resolve to an object", "isError": true } }
]
Silent State Corruption & Stdio Deadlocks — Protocol Violations (CRITICAL)
Read-after-write verification catches tools that reportisError: false without persisting state, as well as hungstdio buffers or severed child-process pipes (with automatic subprocess recycling).
4.
Agent Contract Smells — UX & Consistency Pitfalls (MEDIUM / LOW / INFO)
Explicitly separated from protocol bugs: undocumented glob-onlysearch_files matching, relative vs. parent-joined path contracts, no-upsert skip behavior (create_entities ), and intra-serverisError inconsistencies.
Requires Python >= 3.10.
uvx mcp-crashtest --target sqlite --walks 15 --depth 6
git clone https://github.com/SnowyLumiere/mcp-crashtest.git
cd mcp-crashtest
pip install -e ".[dev]"
Use -- to pass the server command cleanly without option collisions, and --adapter none to run pure zero-shot generic stateful fuzzing with zero domain-specific assumptions:
mcp-crashtest \
--target my-server \
--adapter none \
--sandbox directory \
--seed 42 \
--walks 20 \
--depth 6 \
--fail-on-crash \
--output reports/my_server_report.md \
-- python3 -m my_mcp_server --root {sandbox}
For stateless/API-only servers, use --sandbox none (context-memory rollback and branch exploration still run):
mcp-crashtest \
--target my-api-server \
--adapter none \
--sandbox none \
--walks 20 \
-- node dist/index.js --env test
Requires npx (filesystem / memory) or uvx + git (sqlite / git):
mcp-crashtest --target sqlite --walks 15 --depth 6 --output reports/sqlite_crash_report.md
mcp-crashtest --target git --walks 15 --depth 6 --output reports/git_crash_report.md
mcp-crashtest --target filesystem --walks 15 --depth 6 --output reports/filesystem_crash_report.md
mcp-crashtest --target memory --walks 15 --depth 6 --output reports/memory_crash_report.md
MCP-CrashTest separates the generic zero-shot stateful engine from optional domain contract adapters (mcp_crashtest/adapters/):
| Capability | Generic Mode ( --adapter none ) |
Built-in Domain Adapters ( --adapter auto ) |
|---|---|---|
| Schema Graph Mining | Infers CREATE /READ /MUTATE /DELETE +DATA /STATE /RETRY edges fromtools/list ; handles Pydantic v2anyOf /oneOf /enum andproperties: null |
Adds domain-specific edge bindings for filesystem ,sqlite ,memory ,git |
| Context Flow & Fuzzing | Extracts IDs/keys/URIs/paths from args & JSON responses; injects boundary strings, traversal paths, extreme ints | Adds domain-specific probes (e.g. PK collisions, glob-vs-substring queries, raw * main branch passing) |
| Post-Condition Probes | Probe A ( RETRY_TRAP ), Generic Probe B (Read-After-Write onid /key /uri ), Probe C (SCHEMA_BREAK ), Probe D (STDIO_DEADLOCK ) |
Adds calibrated CONTRACT_SMELL checks (e.g.search_files glob pitfall,describe_table``[] ,create_entities no-upsert skip) |
| Report Provenance | Every finding is tagged with source: generic |
Every adapter finding is explicitly tagged source: adapter:<name> |
--sandbox |
CLI Placeholders | Injected Env Var | Seeded Initial Data |
|---|---|---|---|
directory(default) |
{sandbox} ,{work_dir} |
MCP_SANDBOX_DIR |
sample.txt ,data_dir/config.json ,data_dir/nested_note.txt |
git |
{sandbox} ,{work_dir} |
MCP_SANDBOX_DIR |
Initialized main branch + initial commit (commit.gpgsign=false ) |
sqlite |
{sandbox} ,{db_path} |
MCP_SANDBOX_DB_PATH |
test_target.db with seededusers table (2 rows) |
memory |
{sandbox} ,{memory_path} |
MEMORY_FILE_PATH |
Isolated temp path memory_graph.json |
none |
— | — | No local filesystem sandbox (context rollback still active) |
Verified on 2026-11-20 (--walks 15 --depth 6 --rollbacks 3 --seed 42, Python 3.12, mcp SDK v2.3.0 client with full v1.x/v2.x compatibility; pruned reports in reports/):
| Server | Mode | Protocol Violations | Contract Smells | Headline Minimal Reproducers | Report |
|---|---|---|---|---|---|
mcp-server-sqlite |
--adapter none(pure zero-shot) |
3 ( CRITICAL ) |
0 |
write_query ,create_table ,read_query return"Error: ..." withisError: false (1-step each,source: generic ) |
— |
mcp-server-sqlite |
--adapter auto |
3 ( CRITICAL ) |
1 (MEDIUM ) |
SQL errors return isError: false (×3, 1-step);describe_table returns[] on missing table (1-step) |
sqlite |
mcp-server-git |
--adapter auto |
0 |
4 ( HIGH``SCHEMA_BREAK ) |
git_branch leaks'* main' , breakinggit_checkout ,git_diff ,git_show ,git_merge (2-step each) |
git |
@modelcontextprotocol/server-filesystem |
--adapter auto |
0 |
4 (MEDIUM ) |
list_directory bare filenames vsread_file /read_text_file /get_file_info (2-step);search_files glob-only miss (1-step) |
filesystem |
@modelcontextprotocol/server-memory |
--adapter auto |
0 |
3 (LOW /INFO ) |
create_entities skips existing entities withisError: false (2-step);delete_* idempotent0 differs fromadd_* (1-step) |
memory |
Notes:
- The
sqlitepreset launchesuvx --with "mcp<2" mcp-server-sqlitebecause upstreammcp-server-sqlitecurrently uses themcp 1.x@server.list_resources()API and fails to boot undermcp 2.x.MCP-CrashTestitself supports bothmcpSDK v1.x (inputSchema/isError) and v2.x (input_schema/is_error). reports/MCP_CRASHTEST_COMPREHENSIVE_DIAGNOSIS_20261106.md is preserved as the historical v0.2.0 archive.
- Strict
isErrorSemantics : Never returnisError: falseon SQL errors, constraint violations, or failed mutations. - Cross-Tool I/O Symmetry : Discovery/list tool outputs must be directly consumable by downstream tools without CLI decorations (
* main) or stripped parent paths. - Explicit Matching & Upsert Contracts : Document glob vs. substring matching and provide explicit upsert/conflict errors instead of ambiguous
isError: falseskips. - Actionable Errors & Pipe Backpressure : Return offending parameter names and suggested corrective tools; paginate large responses to prevent
stdiodeadlocks.
| Flag | Default | Description |
|---|---|---|
--target |
"custom" |
Target label or built-in preset ( filesystem ,sqlite ,memory ,git ) |
--adapter |
"auto" |
Domain adapter mode: auto ,none (pure generic), or explicit adapter name(s) |
--cmd /-- <cmd...> |
None |
Server executable (prefer POSIX -- passthrough to avoid flag clashes) |
--sandbox |
"directory" |
directory |sqlite |git |memory |none |
--walks |
15 |
Number of random walk chains |
--depth |
6 |
Max step depth per walk |
--rollbacks |
3 |
Max rollback retries per walk |
--fuzz-ratio |
0.25 |
Boundary fuzzing probability ( 0.0 –1.0 ) |
--timeout |
8.0 |
Per-step & tools/list timeout in seconds |
--init-timeout |
20.0 |
Session initialize() timeout in seconds |
--seed |
None |
Local deterministic RNG seed (does not mutate global random state) |
--lang |
"en" |
Report language: en |zh |
--fail-on-crash |
False |
Exit with code 1 if findings matching--fail-on are detected (for CI gates) |
--fail-on |
"any" |
CI failure scope: any (all findings) |protocol (PROTOCOL_VIOLATION only) |
--output |
"reports/mcp_crash_report.md" |
Output Markdown report path |
--version |
— | Print version and exit |
- Transport : Currently supports**
stdio** transport only.Streamable HTTP / SSE support is on the roadmap. - SQLite Sandbox : In-place binary file rollback assumes per-call SQLite connections (such as
mcp-server-sqlite). Long-lived connection pools with in-memory page caches require session restarts between walks. - Process Memory : For servers keeping unpersisted in-process state, sandbox rollback restores disk state only.
import asyncio
from mcp_crashtest.cli import run_crash_test
async def main():
crashes = await run_crash_test(
cmd="python3",
args=["my_server.py", "--root", "{sandbox}"],
target_name="my-server",
adapter_mode="none",
sandbox_type="directory",
seed=42,
lang="en",
output_report_path="reports/my_server.md",
)
print(f"Hits: {len(crashes)} | Unique: {len({c.fingerprint for c in crashes})}")
asyncio.run(main())
Run the test suite (zero third-party dependencies required for unittest):
PYTHONPATH=. python3 -m unittest discover -s tests -v
PYTHONPATH=. pytest tests/ -v
Released under the MIT License.