{"slug": "show-hn-resilireplay-chaos-testing-for-ai-agents-and-mcp-servers", "title": "Show HN: ResiliReplay - Chaos testing for AI agents and MCP servers", "summary": "ResiliReplay, a new open-source TypeScript toolkit for chaos testing AI agents and MCP servers, lets developers record versioned events, inject seed-controlled faults, and convert broken traces into regression tests. The model-agnostic, local-first tool requires no API key, paid model, Docker, external account, telemetry, or LLM judge, and runs on Node.js 20 or 22 with pnpm. The project, available on GitHub under aliengineering-byte/resilireplay, aims to help developers verify agent recovery from failures deterministically.", "body_md": "**ResiliReplay crash-tests AI agents and MCP servers, replays failures deterministically, and converts broken traces into regression tests.**\n\nIt is a model-agnostic, local-first TypeScript toolkit: record versioned events, inject seed-controlled faults, score recovery from declared evidence, and compile the first causal failure into an editable scenario plus an executable Node test. The deterministic path needs no API key, paid model, Docker, external account, telemetry, or LLM judge.\n\nResiliReplay is defensive testing software. Audit only local or user-owned MCP targets. A report or badge is evidence for one declared suite and version, not a universal security certification.\n\nRequirements: Node.js 20 or 22 and pnpm.\n\n```\ngit clone https://github.com/aliengineering-byte/resilireplay.git\ncd resilireplay\npnpm install --frozen-lockfile\npnpm build\npnpm demo\n```\n\nThe demo runs a real deterministic local subprocess, injects three faults, scores a successful recovery and an unrecovered failure, writes every report format, compiles the failed trace, and executes the generated regression test.\n\nContinue the local tour:\n\n```\npnpm demo:mcp\npnpm exec resilireplay test scenarios\n```\n\nThe MCP demo audits an intentionally vulnerable toy stdio server and a resilient one. The first has two expected safe-canary findings; the second has none. Open `runs/demo/recovered-report/report.html`\n\nin a browser to inspect the standalone recovery report.\n\nSee the [demo guide and verified transcript](/aliengineering-byte/resilireplay/blob/main/docs/DEMO.md) for expected output and asset reproduction.\n\nRecord the bundled deterministic agent and emit a passing baseline report:\n\n```\npnpm exec resilireplay record --output runs/agent/trace.jsonl -- node examples/deterministic-agent/dist/index.js\npnpm exec resilireplay replay --trace runs/agent/trace.jsonl --report-dir runs/agent/report\n```\n\nAdapters for a real agent framework emit the same versioned `TraceEvent`\n\nobjects. See the [adapter guide](/aliengineering-byte/resilireplay/blob/main/docs/ADAPTERS.md); the OpenAI-compatible example is a translation fixture and does not claim to call a live provider.\n\nAudit the bundled resilient server over stdio:\n\n```\npnpm exec resilireplay mcp audit --command \"node examples/resilient-mcp-server/dist/index.js\" --output runs/mcp-audit\n```\n\n`mcp audit`\n\ncaptures `tools/list`\n\nschemas and calls only a tool named `reliability_probe`\n\nby default. Pass `--call-tools`\n\nonly after reviewing tool behavior, because MCP calls may have server-side effects. Non-loopback Streamable HTTP targets also require `--allow-remote`\n\n.\n\nThe [MCP chaos guide](/aliengineering-byte/resilireplay/blob/main/docs/MCP_CHAOS.md) covers controlled faults, transports, authorization, and safe-canary behavior.\n\nApply a built-in fault or reviewable YAML scenario. The same trace plus scenario plus seed produces the same mutation:\n\n```\npnpm exec resilireplay inject --trace runs/agent/trace.jsonl --scenario malformed-json --seed 42 --output runs/agent/failed.jsonl\npnpm exec resilireplay replay --trace runs/agent/failed.jsonl --report-dir runs/agent/failed-report\npnpm exec resilireplay faults\n```\n\nThe replay command above intentionally exits 1: the minimal baseline has no recovery event after the injected malformed response. It still writes the failed report bundle for review. Use `pnpm exec resilireplay test scenarios`\n\nwhen a CI command should verify both expected-pass and expected-failure scenarios with exit 0.\n\nProvider and transport faults include bounded latency, timeout, 429/5xx, reset, truncation, malformed JSON, duplicates, and stale responses. Tool and workflow faults cover errors, permissions, disposable missing files, corrupt results, side-effect duplication, handoff loss, wrong recipients, stale state, conflicting instructions, and loops.\n\nSee [custom fault scenarios](/aliengineering-byte/resilireplay/blob/main/docs/CUSTOM_FAULTS.md) for the YAML contract and safety rules.\n\n`pnpm demo`\n\ncreates a failed trace. Convert it into a minimized fixture, scenario, manifest, and executable `node:test`\n\n:\n\n```\npnpm exec resilireplay generate-test --trace runs/demo/failed.jsonl --output runs/generated-regression\n```\n\nThe generated test executes immediately by default. `manifest.json`\n\nlinks the source trace, minimized fixture, scenario, and test with SHA-256 hashes.\n\n``` php\nflowchart LR\n    A[\"Record JSONL trace\"] --> B[\"Inject deterministic fault\"]\n    B --> C[\"Replay\"]\n    C --> D[\"Score recovery\"]\n    D --> E[\"Minimize causal failure\"]\n    E --> F[\"Generate regression\"]\n    F --> C\n```\n\nThis excerpt is captured from the no-key demo:\n\n```\nResiliReplay v0.1.0  PASS\nRecovery score  100/100\nCompletion      yes\nRecovery        safe\nRetries         1/3\nSafety          compliant\n```\n\nEach run can emit:\n\n| Format | File | Intended use |\n|---|---|---|\n| Terminal | `terminal.txt` |\nLocal and CI logs |\n| JSON | `report.json` |\nAutomation and dashboards |\n| HTML | `report.html` |\nStandalone human review |\n| JUnit | `junit.xml` |\nTest runners and CI annotations |\n| SARIF 2.1.0 | `report.sarif` |\nCode-scanning ingestion |\n| Manifest | `run-manifest.json` |\nArtifact and metrics hashes |\n| SVG | `badge.svg` |\nScope-limited run status |\n\nSee the [report schema](/aliengineering-byte/resilireplay/blob/main/docs/REPORT_SCHEMA.md).\n\nThe stable boundary is a strict, provider-neutral `TraceEvent`\n\n, not a model SDK:\n\n| Package | Responsibility |\n|---|---|\n`@resilireplay/core` |\nEvents, redaction, fault engine, deterministic scoring, path safety |\n`@resilireplay/trace` |\nCanonical JSONL and failed-trace-to-regression compiler |\n`@resilireplay/reporters` |\nTerminal, JSON, HTML, JUnit, SARIF, manifests, badges |\n`@resilireplay/mcp-chaos` |\nAuthorized MCP discovery, calling, mutation, canaries, evidence |\n`@resilireplay/proxy` |\nLoopback provider/transport mutation proxy |\n`resilireplay` |\nCross-platform CLI, recording, replay, scenarios, cleanup |\n`@resilireplay/github-action` |\nRepository-native CI integration |\n\nRead the [architecture](/aliengineering-byte/resilireplay/blob/main/docs/ARCHITECTURE.md) for invariants and package dependencies.\n\n`record`\n\nexecutes the exact command you supply; it is not an OS sandbox.- Audit only MCP servers you own or are authorized to test.\n- Review MCP schemas before\n`--call-tools`\n\n; tools can have side effects. - Omit secrets at the adapter source even though credential-shaped values and sensitive keys are redacted before trace storage.\n- Treat reports as sensitive test evidence if source traces contain private application data.\n\nFilesystem faults use owned temporary directories, output paths are containment-checked, spawned processes have deadlines and cleanup, and listeners bind to loopback unless a caller deliberately chooses another host.\n\nRead [SECURITY.md](/aliengineering-byte/resilireplay/blob/main/SECURITY.md) and [THREAT_MODEL.md](/aliengineering-byte/resilireplay/blob/main/THREAT_MODEL.md) before running untrusted commands or remote MCP targets.\n\n- Streamable HTTP support has less integration coverage than stdio.\n`record`\n\nis not an OS sandbox.- MCP tool calls may have server-side effects.\n- Causal minimization is strongest when adapters provide\n`parentId`\n\nand`causeId`\n\n. - Streaming provider output is aggregated into response events in v0.1.0.\n- Report hashes prove linkage and integrity, not authenticity; manifests are unsigned.\n- Packages are not published to npm.\n\nSee [known limitations](/aliengineering-byte/resilireplay/blob/main/docs/LIMITATIONS.md) and the [roadmap](/aliengineering-byte/resilireplay/blob/main/docs/ROADMAP.md).\n\nRun the complete local gate:\n\n```\npnpm quality\n```\n\nUse the composite action:\n\n```\n- uses: aliengineering-byte/resilireplay@v0.1.0\n  with:\n    scenarios: scenarios\n```\n\nResiliReplay is Apache-2.0 licensed. Contributions should be deterministic, bounded, and covered by tests. See [CONTRIBUTING.md](/aliengineering-byte/resilireplay/blob/main/CONTRIBUTING.md), [CODE_OF_CONDUCT.md](/aliengineering-byte/resilireplay/blob/main/CODE_OF_CONDUCT.md), the [launch reference](/aliengineering-byte/resilireplay/blob/main/docs/LAUNCH.md), [release evidence](/aliengineering-byte/resilireplay/blob/main/RELEASE_EVIDENCE.md), and the [v0.1.0 release](https://github.com/aliengineering-byte/resilireplay/releases/tag/v0.1.0).\n\nBuilt and maintained by **Ali**.", "url": "https://wpnews.pro/news/show-hn-resilireplay-chaos-testing-for-ai-agents-and-mcp-servers", "canonical_source": "https://github.com/aliengineering-byte/resilireplay", "published_at": "2026-07-31 01:44:29+00:00", "updated_at": "2026-07-31 01:52:22.716577+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["ResiliReplay", "aliengineering-byte", "GitHub", "Node.js", "pnpm", "MCP"], "alternates": {"html": "https://wpnews.pro/news/show-hn-resilireplay-chaos-testing-for-ai-agents-and-mcp-servers", "markdown": "https://wpnews.pro/news/show-hn-resilireplay-chaos-testing-for-ai-agents-and-mcp-servers.md", "text": "https://wpnews.pro/news/show-hn-resilireplay-chaos-testing-for-ai-agents-and-mcp-servers.txt", "jsonld": "https://wpnews.pro/news/show-hn-resilireplay-chaos-testing-for-ai-agents-and-mcp-servers.jsonld"}}