{"slug": "show-hn-dynobox-integration-tests-for-agents-using-cli-and-mcp-mocks", "title": "Show HN: Dynobox – Integration tests for agents using CLI and MCP mocks", "summary": "Dynobox, an open-source integration test runner for coding agents, launched with support for running the same scenario across Claude Code, OpenAI Codex, OpenCode 2.x, Pi, Cursor CLI, and Google Antigravity CLI 1.1.14+. Dynobox asserts on tool calls, commands, files, HTTP requests, transcripts, and final answers without requiring one model to judge another, and it replaces bare executable calls with scenario-scoped static, sequential, or handler responses through experimental CLI mocks plus HTTP MCP tool fixtures checked via MCP mocks. The tool requires Node.js 22+ and at least one installed, authenticated harness on PATH, and its authors warn that temporary work directories separate job files but are not security sandboxes.", "body_md": "Open-source test runner for coding agents and skills: run real workflows, capture evidence, assert on tools, commands, files, and answers.\n\n[Get started](https://docs.dynobox.xyz/getting-started) ·\n  [How it works](https://docs.dynobox.xyz/how-it-works) ·\n  [Documentation](https://docs.dynobox.xyz) ·\n  [Dashboard](https://dash.dynobox.xyz) ·\n  [Agent skills](https://www.skills.sh/dynobox/skills) ·\n  [Examples](https://github.com/dynobox/examples)\n\n- Assert on tool calls, commands, files, HTTP requests, transcripts, and final answers without requiring one model to judge another.\n- Run the same scenario through Claude Code, OpenAI Codex, OpenCode, Pi, Cursor CLI, and Google Antigravity CLI to find behavior that varies between environments.\n- Test multi-step tasks in fresh temporary work directories, repeat them to expose flaky behavior, and keep the evidence when something fails.\n- Replace bare executable calls with scenario-scoped static, sequential, or\nhandler responses using [experimental CLI mocks](https://docs.dynobox.xyz/config-authoring/#cli-mocks) .\n- Provide HTTP MCP tool fixtures and check received calls with\n[MCP mocks](https://github.com/dynobox/dynobox/blob/main/docs/mcp-mocking.md) .\n\nNode.js 22+ and at least one supported harness installed, authenticated, and\navailable on `PATH`: Claude Code (` claude`), Codex (` codex`), OpenCode 2.x\n(`opencode`), Pi (` pi`), Cursor CLI (` cursor-agent`), or Google Antigravity CLI\n1.1.14+ (`agy`).\n\n```\nnpx dynobox@latest init                         # Claude Code (default)\nnpx dynobox@latest init --harness codex         # OpenAI Codex\nnpx dynobox@latest init --harness opencode      # OpenCode\nnpx dynobox@latest init --harness pi            # Pi\nnpx dynobox@latest init --harness cursor        # Cursor CLI\nnpx dynobox@latest init --harness antigravity   # Google Antigravity CLI\nnpx dynobox run\n```\n\nRun one `init` command for a harness that is installed and authenticated, then\nrun the generated dyno.\n\n`dynobox init` creates a starter dyno in `dynobox/example.dyno.mjs`.\n`dynobox run` discovers `*.dyno.{mjs,js,ts,mts,yaml,yml}` files below the current\ndirectory and runs their scenarios against the configured harnesses.\n\nOnly run dynos you trust. JavaScript and TypeScript configs are imported, and setup and verification commands execute on your machine. Temporary work directories separate job files, but they are not security sandboxes; processes can access the host according to their permissions.\n\nA dyno combines the prompt, fixture setup, harnesses, and acceptance criteria in one TypeScript, JavaScript, or YAML file. Include only harnesses installed and authenticated in the current environment:\n\n```\n# package.dyno.yaml\nname: package-script\nharnesses:\n  - claude-code\n  - codex\n  - opencode\n  - pi\n  - cursor\n  - antigravity\nscenarios:\n  - name: detects the test script\n    setup:\n      - |\n        cat > package.json <<'JSON'\n        {\"scripts\":{\"test\":\"vitest run\"}}\n        JSON\n    prompt: >-\n      Use cat package.json and tell me whether this project has a test script.\n    assertions:\n      - type: command.called\n        executable: cat\n        command: {args: [package.json]}\n      - type: tool.notCalled\n        tool: edit_file\n      - type: artifact.contains\n        path: package.json\n        text: vitest run\n      - type: finalMessage.contains\n        text: test\n```\n\nRun the test:\n\n```\nnpx dynobox run package.dyno.yaml\n```\n\nThe same shape works in TypeScript and JavaScript via\n[`@dynobox/sdk`](https://www.npmjs.com/package/@dynobox/sdk)\n(`defineDyno`, `tool`, `command`, ...). See\n[Config Authoring](https://docs.dynobox.xyz/config-authoring).\n\nDynobox launches the selected harnesses in fresh temporary work directories, records the observable behavior, and evaluates each assertion against captured evidence. When a check fails, the output shows what was expected and what was observed.\n\n| Evidence | Example checks | What it answers | \n|---|---|---|\n| Tools | `tool.called` ,`tool.notCalled` | Did the agent use or avoid a tool? | \n| Commands | `command.called` ,`command.notCalled` | Did it execute the expected shell command? | \n| Files | `artifact.exists` ,`artifact.contains` ,`artifact.unchanged` | Did it create, change, or preserve the right files? | \n| Skills | `skill.referenced` | Did it reference the required skill instructions? | \n| Network* | `http.called` ,`http.notCalled` | Did a proxy-aware child process call the expected endpoint? | \n| Response | `transcript.contains` ,`finalMessage.contains` | Did the interaction contain required information? | \n| Logic | `sequence.inOrder` ,`anyOf` | Did the observed behavior follow an accepted path or order? | \n| Verification | `verify.command` | Does the completed work pass a custom executable check? | \n\n* HTTP capture observes child-process traffic that honors Dynobox's proxy and\nCA settings. Harness-native web tools and clients with independent networking\nmay not be captured. [Learn how HTTP capture\nworks](https://docs.dynobox.xyz/how-it-works#4-capture-observable-evidence).\n\nSee the [assertion reference](https://docs.dynobox.xyz/config-authoring/#assertions)\nfor every matcher and authoring option.\n\nOverride the configured harnesses from the command line:\n\n```\nnpx dynobox run --harness claude-code\nnpx dynobox run --harness codex\nnpx dynobox run --harness opencode\nnpx dynobox run --harness pi\nnpx dynobox run --harness cursor\nnpx dynobox run --harness antigravity\nnpx dynobox run --harness claude-code,codex,opencode,pi,cursor,antigravity\n```\n\nRepeat every scenario and harness pair to measure pass rates:\n\n```\nnpx dynobox run \\\n  --harness claude-code,codex,opencode,pi,cursor,antigravity \\\n  --iterations 5\n```\n\nFor each scenario, Dynobox runs the selected harness configurations in parallel. Repeated iterations for one harness configuration remain sequential, and the next scenario starts after every harness for the current scenario finishes.\n\nRepeated runs render compact pass-rate rows such as `.FF..` while retaining the\nfailed iteration evidence for diagnosis.\n\nDynobox runs locally and works without an account. Use terminal output for\ndevelopment, `--reporter json` for automation, or authenticate with\n`dynobox login` and add `--save-run` to publish a compact run summary to the\n[Dynobox dashboard](https://dash.dynobox.xyz). Set `DYNOBOX_UPLOAD_URL` to post\nthe same payload to your own endpoint without sending Dynobox credentials.\n\nSaved-run data is length-capped but not redacted. It includes available Git\nidentity and revision metadata. All jobs can include authored assertion data and\nmatched evidence such as requested endpoint URLs, tool commands, and\nverification output. Failed jobs can additionally include command or harness\ndiagnostics. Do not use `--save-run` when those values should not be shared.\n\nDynobox is closer to unit testing than model-graded evals. It evaluates captured behavior—tool calls, shell commands, files, and HTTP requests—using explicit, repeatable assertions.\n\nCompared with one-off scripts, Dynobox provides normalized assertions, fresh temporary work directories, captured failure evidence, and iteration pass rates for measuring flaky behavior.\n\n| Package | Description | \n|---|---|\n| [`dynobox`](https://www.npmjs.com/package/dynobox) | CLI for discovering and running dynos | \n| [`@dynobox/sdk`](https://www.npmjs.com/package/@dynobox/sdk) | Type-safe helpers for authoring dynos | \n\nDynobox is early-access software. The CLI, SDK, and report formats may evolve before 1.0.\n\nSee [CONTRIBUTING.md](https://github.com/dynobox/dynobox/blob/main/CONTRIBUTING.md) to run the monorepo locally and propose\na change.\n\nDynobox is [Apache-2.0 licensed](https://github.com/dynobox/dynobox/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-dynobox-integration-tests-for-agents-using-cli-and-mcp-mocks", "canonical_source": "https://github.com/dynobox/dynobox", "published_at": "2026-11-19 19:30:58+00:00", "updated_at": "2026-11-19 19:45:59.999546+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols", "mlops"], "entities": ["Dynobox", "Claude Code", "OpenAI Codex", "OpenCode", "Pi", "Cursor CLI", "Google Antigravity CLI", "@dynobox/sdk"], "also_reported_by": [], "takeaway": "Dynobox lets builders run one scenario across Claude Code, Codex, OpenCode, Pi, Cursor CLI, and Antigravity CLI and assert on tool calls, commands, files, and final answers without an LLM judge, using CLI and MCP mocks for deterministic fixtures.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/show-hn-dynobox-integration-tests-for-agents-using-cli-and-mcp-mocks", "markdown": "https://wpnews.pro/news/show-hn-dynobox-integration-tests-for-agents-using-cli-and-mcp-mocks.md", "text": "https://wpnews.pro/news/show-hn-dynobox-integration-tests-for-agents-using-cli-and-mcp-mocks.txt", "jsonld": "https://wpnews.pro/news/show-hn-dynobox-integration-tests-for-agents-using-cli-and-mcp-mocks.jsonld"}}