ResiliReplay crash-tests AI agents and MCP servers, replays failures deterministically, and converts broken traces into regression tests.
It is a model-agnostic, local-first TypeScript toolkit: record versioned events, inject seed-controlled faults, score recovery from declared evidence, and compile the first causal failure into an editable scenario plus an executable Node test. The deterministic path needs no API key, paid model, Docker, external account, telemetry, or LLM judge.
ResiliReplay is defensive testing software. Audit only local or user-owned MCP targets. A report or badge is evidence for one declared suite and version, not a universal security certification.
Requirements: Node.js 20 or 22 and pnpm.
git clone https://github.com/aliengineering-byte/resilireplay.git
cd resilireplay
pnpm install --frozen-lockfile
pnpm build
pnpm demo
The demo runs a real deterministic local subprocess, injects three faults, scores a successful recovery and an unrecovered failure, writes every report format, compiles the failed trace, and executes the generated regression test.
Continue the local tour:
pnpm demo:mcp
pnpm exec resilireplay test scenarios
The MCP demo audits an intentionally vulnerable toy stdio server and a resilient one. The first has two expected safe-canary findings; the second has none. Open runs/demo/recovered-report/report.html
in a browser to inspect the standalone recovery report.
See the demo guide and verified transcript for expected output and asset reproduction.
Record the bundled deterministic agent and emit a passing baseline report:
pnpm exec resilireplay record --output runs/agent/trace.jsonl -- node examples/deterministic-agent/dist/index.js
pnpm exec resilireplay replay --trace runs/agent/trace.jsonl --report-dir runs/agent/report
Adapters for a real agent framework emit the same versioned TraceEvent
objects. See the adapter guide; the OpenAI-compatible example is a translation fixture and does not claim to call a live provider.
Audit the bundled resilient server over stdio:
pnpm exec resilireplay mcp audit --command "node examples/resilient-mcp-server/dist/index.js" --output runs/mcp-audit
mcp audit
captures tools/list
schemas and calls only a tool named reliability_probe
by default. Pass --call-tools
only after reviewing tool behavior, because MCP calls may have server-side effects. Non-loopback Streamable HTTP targets also require --allow-remote
.
The MCP chaos guide covers controlled faults, transports, authorization, and safe-canary behavior.
Apply a built-in fault or reviewable YAML scenario. The same trace plus scenario plus seed produces the same mutation:
pnpm exec resilireplay inject --trace runs/agent/trace.jsonl --scenario malformed-json --seed 42 --output runs/agent/failed.jsonl
pnpm exec resilireplay replay --trace runs/agent/failed.jsonl --report-dir runs/agent/failed-report
pnpm exec resilireplay faults
The replay command above intentionally exits 1: the minimal baseline has no recovery event after the injected malformed response. It still writes the failed report bundle for review. Use pnpm exec resilireplay test scenarios
when a CI command should verify both expected-pass and expected-failure scenarios with exit 0.
Provider and transport faults include bounded latency, timeout, 429/5xx, reset, truncation, malformed JSON, duplicates, and stale responses. Tool and workflow faults cover errors, permissions, disposable missing files, corrupt results, side-effect duplication, handoff loss, wrong recipients, stale state, conflicting instructions, and loops.
See custom fault scenarios for the YAML contract and safety rules.
pnpm demo
creates a failed trace. Convert it into a minimized fixture, scenario, manifest, and executable node:test
:
pnpm exec resilireplay generate-test --trace runs/demo/failed.jsonl --output runs/generated-regression
The generated test executes immediately by default. manifest.json
links the source trace, minimized fixture, scenario, and test with SHA-256 hashes.
flowchart LR
A["Record JSONL trace"] --> B["Inject deterministic fault"]
B --> C["Replay"]
C --> D["Score recovery"]
D --> E["Minimize causal failure"]
E --> F["Generate regression"]
F --> C
This excerpt is captured from the no-key demo:
ResiliReplay v0.1.0 PASS
Recovery score 100/100
Completion yes
Recovery safe
Retries 1/3
Safety compliant
Each run can emit:
| Format | File | Intended use |
|---|---|---|
| Terminal | terminal.txt |
|
| Local and CI logs | ||
| JSON | report.json |
|
| Automation and dashboards | ||
| HTML | report.html |
|
| Standalone human review | ||
| JUnit | junit.xml |
|
| Test runners and CI annotations | ||
| SARIF 2.1.0 | report.sarif |
|
| Code-scanning ingestion | ||
| Manifest | run-manifest.json |
|
| Artifact and metrics hashes | ||
| SVG | badge.svg |
|
| Scope-limited run status |
See the report schema.
The stable boundary is a strict, provider-neutral TraceEvent
, not a model SDK:
| Package | Responsibility |
|---|---|
@resilireplay/core |
|
| Events, redaction, fault engine, deterministic scoring, path safety | |
@resilireplay/trace |
|
| Canonical JSONL and failed-trace-to-regression compiler | |
@resilireplay/reporters |
|
| Terminal, JSON, HTML, JUnit, SARIF, manifests, badges | |
@resilireplay/mcp-chaos |
|
| Authorized MCP discovery, calling, mutation, canaries, evidence | |
@resilireplay/proxy |
|
| Loopback provider/transport mutation proxy | |
resilireplay |
|
| Cross-platform CLI, recording, replay, scenarios, cleanup | |
@resilireplay/github-action |
|
| Repository-native CI integration |
Read the architecture for invariants and package dependencies.
record
executes the exact command you supply; it is not an OS sandbox.- Audit only MCP servers you own or are authorized to test.
- Review MCP schemas before
--call-tools
; tools can have side effects. - Omit secrets at the adapter source even though credential-shaped values and sensitive keys are redacted before trace storage.
- Treat reports as sensitive test evidence if source traces contain private application data.
Filesystem faults use owned temporary directories, output paths are containment-checked, spawned processes have deadlines and cleanup, and listeners bind to loopback unless a caller deliberately chooses another host.
Read SECURITY.md and THREAT_MODEL.md before running untrusted commands or remote MCP targets.
- Streamable HTTP support has less integration coverage than stdio.
record
is not an OS sandbox.- MCP tool calls may have server-side effects.
- Causal minimization is strongest when adapters provide
parentId
andcauseId
. - Streaming provider output is aggregated into response events in v0.1.0.
- Report hashes prove linkage and integrity, not authenticity; manifests are unsigned.
- Packages are not published to npm.
See known limitations and the roadmap.
Run the complete local gate:
pnpm quality
Use the composite action:
- uses: aliengineering-byte/resilireplay@v0.1.0
with:
scenarios: scenarios
ResiliReplay is Apache-2.0 licensed. Contributions should be deterministic, bounded, and covered by tests. See CONTRIBUTING.md, CODE_OF_CONDUCT.md, the launch reference, release evidence, and the v0.1.0 release.
Built and maintained by Ali.