{"slug": "firedrill-stateful-tool-simulation-for-ai-agents", "title": "Firedrill: Stateful tool simulation for AI agents", "summary": "Firedrill released a simulation and testing framework for AI agents that lets developers define synthetic stateful tools over HTTP, MCP, CLI, and function bindings and assert on tool calls, state changes, and events. The framework ships as the Python package firedrill-run (Python 3.10+) and the npm packages @firedrill-run/cli and @firedrill-run/sdk, requiring Node.js 20.19 or later, and offers scenario-based tests with faults, response overrides, virtual time, isolated world state, seeded data, snapshots, resets, and HTML, JSON, and JUnit reports. Tools install on demand from a catalog as @firedrill-tools/<id> dev dependencies, with the CLI verifying package integrity before installation consent.", "body_md": "Firedrill is a simulation and testing framework for AI agents. Define synthetic tools and data, run your agent against them, and assert on tool calls, state changes, and events.\n\n- Stateful tools with HTTP, MCP, CLI, and function bindings.\n- Scenario-based tests with faults, response overrides, and virtual time.\n- Isolated world state, seeded data, snapshots, and resets.\n- HTML, JSON, and JUnit reports with timelines and optional browser captures.\n- Repository-defined tools, including independently distributed packages.\n\n[Quickstart](#quickstart) · [Python](https://github.com/firedrill-tools/firedrill/blob/main/python/README.md) · [TypeScript](#using-the-sdk) ·\n[Documentation](https://docs.firedrill.run) · [Neutral example](https://github.com/firedrill-tools/firedrill/blob/main/examples/quickstart/README.md) ·\n[Gmail Agent example](https://github.com/firedrill-tools/firedrill-example-gmail-agent)\n\nInstall the current release from [PyPI](https://pypi.org/project/firedrill-run/)\ninto your virtual environment with Python 3.10 or later:\n\n```\npython -m pip install \"firedrill-run[pytest]\"\nfiredrill --help\n```\n\nImport the local SDK with `from firedrill import World, run_drills`. The wheel\nincludes the runtime, CLI, inspector, and report engine. No separate Node.js or\nnpm installation is required. Choose Tools with `firedrill init`; they install\non demand and use the same packages as TypeScript. See the\n[Python guide](https://github.com/firedrill-tools/firedrill/blob/main/python/README.md) for pytest, async agents, mocks, and browser tests.\n\nRequires Node.js 20.19 or later.\n\nInstall the current release in your project:\n\n```\nnpm install --save-dev @firedrill-run/cli @firedrill-run/sdk\nnpx firedrill --help\n```\n\nOr run the CLI without adding a project dependency:\n\n```\nnpx @firedrill-run/cli init\n```\n\nFor a source checkout, use pnpm 9.15–10:\n\n```\npnpm install --frozen-lockfile\npnpm build\n\n# Use the built CLI in this terminal.\nexport FIREDRILL_CLI=\"$PWD/packages/cli/dist/bin.js\"\nfiredrill() { node \"$FIREDRILL_CLI\" \"$@\"; }\n```\n\nThe examples below use `npx firedrill` inside a project that has\n`@firedrill-run/cli` installed. Never run bare `npx firedrill` elsewhere: outside\nsuch a project npm resolves an unrelated package with that name. Python\ninstallations expose `firedrill` directly, so drop the `npx` prefix. From source,\nuse the shell function above or invoke the CLI directly with\n`node /path/to/firedrill/packages/cli/dist/bin.js`.\n\nThe programmatic API is `@firedrill-run/sdk`. To prepare installable archives of the\nCLI, SDK, and other packages from this checkout, run\n`pnpm pack:artifacts -- --output /absolute/path/to/an/empty/directory`.\nThe output includes a package manifest.\n\nCreate a project with a synthetic record store:\n\n```\nmkdir firedrill-example\ncd firedrill-example\nnpm init -y\nnpm install --save-dev @firedrill-run/cli\nnpx firedrill init --custom records\nnpx firedrill serve\n```\n\n`init` creates a Tool declaration, a behavior module, and starting data.\n`serve` starts the backend and opens the inspector.\n\nOpen **Tools** to inspect the implementation or call an operation.\n**State & activity** shows records and calls; **Connect agent** provides the\nconnection settings. Tools with a bundled UI also have an **Open app** action.\nBrowser actions and API calls use the same state.\n\nThe server listens on loopback using available ports. Keep the terminal open;\nCtrl+C stops it. Use `--no-open` to skip opening the inspector automatically.\n\nRun `firedrill init` in an existing project for guided setup, or select one of\nthe published packages in the [Tool catalog](https://github.com/firedrill-tools/firedrill/blob/main/registry/README.md), for example\n`npx firedrill init --tool gmail --install` or\n`npx firedrill tool add gmail --install`. Both add the catalog's exact\n`@firedrill-tools/<id>` npm release as an ordinary dev dependency and refuse an\narchive whose integrity differs from the catalog. The CLI shows the exact\npackage and version before it asks for installation consent.\n\nTools can run independently of tests. To check an agent's behavior, add a drill.\n\nFor a model-backed project, see the\n[Gmail Agent example](https://github.com/firedrill-tools/firedrill-example-gmail-agent):\nan existing Claude Agent SDK assistant runs three drills against a pinned\nstateful Gmail Tool, with its synthetic mailbox and reports kept in the project.\n\nA **drill** defines an agent task, starting conditions, and assertions about the\nresult. To try one, stop the server, return to the parent directory (`cd ..`),\nand create the example test project:\n\n```\nmkdir firedrill-first-drill\ncd firedrill-first-drill\nnpm init -y\nnpm install --save-dev @firedrill-run/cli\nnpx firedrill init --path template\nnpx firedrill validate\nnpx firedrill plan\nnpx firedrill run changes-resource\nnpx firedrill inspect\n```\n\nThe template contains a deterministic example agent that writes `7` to a record.\nIts drill checks that the write succeeded once and that the final value is `7`.\nReplace the example target with your agent when adding your own tests.\n\nTo inspect a failing result, edit\n`firedrill/drills/changes-resource.drill.yaml`: change the `value-changed`\nassertion's expected value to `8`, keeping `task.input.value` at `7`.\nRerun the drill. The report shows expected `8` and actual `7`; the process exits\nwith code `1`. Restore the expectation afterwards.\n\n| Command | Purpose | \n|---|---|\n| `firedrill validate` | Check source definitions | \n| `firedrill plan` | List tools, scenarios, targets, and drills | \n| `firedrill run <id>` | Run one drill | \n| `firedrill` | Run all drills | \n| `firedrill serve` | Start a standalone synthetic backend | \n| `firedrill inspect` | Browse definitions and saved results | \n| `firedrill ci init github` | Add drills to pull requests, pushes, schedules, or manual CI jobs | \n\nRun the same repository-owned drill command locally and in CI. A pull request is one useful trigger, not a requirement: pushes, scheduled suites, manual jobs, and other CI providers use the same command and reports.\n\n```\nfiredrill ci describe\nfiredrill ci init github\n```\n\nThe initializer detects the project's package manager or Python setup, preserves\nan existing `test:firedrill`/` drills` script when present, and writes\n`.github/workflows/firedrill.yml`. It uploads self-contained HTML, JSON, and JUnit\nreports even when a drill fails. Use `--run` for a caller-owned SDK harness and\nrepeat `--trigger` to select exact events. See the\n[continuous-integration guide](https://github.com/firedrill-tools/firedrill/blob/main/docs/continuous-integration.md).\n\nAdd a focused safety suite to pull requests without moving the agent into Firedrill or changing production agent code:\n\n```\nfiredrill ci init github \\\n  --run \"npm run test:firedrill\" \\\n  --trigger pull-request \\\n  --trigger manual\n```\n\nCommit the generated workflow. GitHub runs the repository's existing command at\nthe exact pull-request revision and shows its exit status as a normal check.\nFiredrill retains `.firedrill/reports/` as an Actions artifact even when a drill\nfails, so a reviewer can open the HTML result, JUnit output, changed state, and\nordered evidence behind the verdict. Requiring that check in branch protection\nis optional.\n\nThe hosted workflow adds managed worlds, a base-versus-head behavior comparison, a durable evidence link, and a Firedrill check and pull-request summary. Local CI remains complete and account-free.\n\n| Term | Meaning | \n|---|---|\n| Tool | A synthetic dependency with callable operations, input/output schemas, and an implementation | \n| World | Tools, starting data, identities, permissions, and a clock | \n| Scenario | A variation of the starting data, permissions, faults, or scheduled events | \n| Target | Configuration for invoking the agent under test | \n| Drill | A task and its assertions | \n| Run | A recorded execution result, including checks, calls, and state changes | \n\nActors identify who is calling a tool and which operations they may use.\nPersonas provide descriptions of those identities. See\n[people and permissions](https://github.com/firedrill-tools/firedrill/blob/main/docs/world-authoring.md#people-and-permissions).\n\nConfigure the agent's dependencies in test setup:\n\n| Dependency | Integration | \n|---|---|\n| HTTP client | Point its base URL and authentication at the Tool's declared HTTP routes | \n| MCP server | Use the world's MCP endpoint and actor token | \n| CLI tool | Use a test-side command adapter or the Firedrill world CLI | \n| Function or SDK method | Use `mock_tool` with Python's`unittest.mock` / pytest, or`mockTool` with JavaScript runner mocks | \n| Web interface | Use a Playwright harness or the optional browser-test package | \n\nBindings use existing configuration or test-side adapters, leaving production\nagent logic unchanged. Hardcoded dependencies need an interceptable boundary or\nan explicit adapter. See [binding recipes](https://github.com/firedrill-tools/firedrill/blob/main/docs/quickstart.md#3-keep-the-agent-integration-at-one-seam)\nand [test-side mocking](https://github.com/firedrill-tools/firedrill/blob/main/docs/test-mocking.md).\n\nTargets can invoke a module, start a command, call an HTTP endpoint, or use an\n`external` callback supplied by a test harness. External targets run through the\nSDK; module, command, and HTTP targets can also run through the CLI.\n\nModel credentials belong to the agent process. For command targets, pass model\ncredentials and other required host variables through `environmentFromHost`.\nSet target timeouts for the complete model/tool loop.\n\nUse `runDrills` from an existing test runner:\n\n``` js\nimport { runDrills } from \"@firedrill-run/sdk\";\n\nconst result = await runDrills({\n  root: process.cwd(),\n  drill: \"my-drill\",\n  agent: ({ task, binding, signal }) =>\n    runMyAgent({ task, environment: binding.environment, signal }),\n});\n\nexpect(result.verdict).toBe(\"passed\");\n```\n\nThis example assumes a declared `my-drill` with an `external` target.\n`runMyAgent` is your test adapter; it applies the supplied connection values to\nyour agent. `expect` comes from your test runner.\n\n`runDrills({ setup })` supports per-test data, fault, and Tool overrides.\n`createLocalWorld()` provides direct control over calls, state, time, and resets.\nSee the [SDK reference](https://github.com/firedrill-tools/firedrill/blob/main/packages/sdk/README.md) for lifecycle hooks, concurrency,\ncapture, and report APIs.\n\nThe example template uses the following layout:\n\n```\nyour-project/\n  firedrill.json                         # source location and selected packages\n  firedrill/\n    world.yaml                          # starting data, identities, access, time\n    tools/resource-store/\n      resource-store.tool.yaml          # operations and state schemas\n      behavior.mjs                      # operation implementations\n    scenarios/baseline.scenario.yaml    # starting conditions\n    targets/starter-agent.target.yaml  # agent invocation\n    drills/changes-resource.drill.yaml  # task and assertions\n    suites/resource-store-conformance.suite.yaml\n  firedrill-example/agent.mjs           # example agent\n  .firedrill-tools/                     # vendored Tool dependencies\n  .firedrill/                           # generated state, builds, and reports\n```\n\nDefinitions support JSON or YAML; use either consistently or mix them.\nResource suffixes identify file types, such as `.tool.json` and `.drill.yaml`.\nReferences use IDs inside the files, so you can organize folders as needed.\nTool implementations are JavaScript or TypeScript.\n\nCommit definitions, behavior modules, test code, package manifests, lockfiles,\nand referenced `.firedrill-tools/` archives. Generated files under `.firedrill/`\nare ignored by `init`. Keep credentials, `.env` files, and `node_modules/` ignored\nas well.\n\nTool state lives in SQLite. Runtime writes leave the source definitions unchanged.\nSource edits apply to the next build; restart `serve` to load them.\n\nEach trial or retry uses an isolated world. A full reset restores its baseline; a scoped reset restores selected Tools. Reset affects the synthetic environment, not the agent's own database or memory. Saved reports are retained.\n\nSee [source authoring](https://github.com/firedrill-tools/firedrill/blob/main/docs/world-authoring.md) and\n[world controls](https://github.com/firedrill-tools/firedrill/blob/main/docs/local-world-control.md).\n\nOpen `.firedrill/reports/index.html` to browse saved runs, or use **Results**\nin the inspector. Each run directory contains:\n\n- An HTML report with task, outcome, expected/actual checks, and tool activity.\n- JSON results, JUnit XML, and ordered evidence.\n- An integrity manifest and any captured attachments.\n\n```\nnpx firedrill compare .firedrill/reports/<baseline> .firedrill/reports/<candidate>\nnpx firedrill report verify .firedrill/reports/<run-id>\n```\n\nReports distinguish assertion failures, execution errors, and incomplete evidence.\nCLI exit codes are `0` for a passing selection, `1` for source/execution/check\nfailure, and `2` for invalid command usage.\n\nThe reproduction command pins the build and seed. This reproduces the world inputs; live model responses can still vary. Comparisons identify changed inputs before presenting result differences.\n\nKeep attachment folders with their reports. Review captured data before sharing.\nBundle verification checks integrity, not authorship. See\n[running and results](https://github.com/firedrill-tools/firedrill/blob/main/docs/running-and-results.md) for the report layout and CI use.\n\n| Capability | Usage or guide | \n|---|---|\n| Repeated trials | `npx firedrill run changes-resource --trials 3 --seed 42` | \n| Filtered, concurrent execution | `npx firedrill run --tag smoke --concurrency 4` | \n| Watch mode | `npx firedrill run changes-resource --watch` | \n| Per-test data and Tool overrides | [SDK setup](https://github.com/firedrill-tools/firedrill/blob/main/packages/sdk/README.md#per-test-synthetic-data-and-tools) | \n| State inspection, virtual time, and resets | [World controls](https://github.com/firedrill-tools/firedrill/blob/main/docs/local-world-control.md) | \n| Multi-interaction simulations | [Drill timelines](https://github.com/firedrill-tools/firedrill/blob/main/docs/running-and-results.md#repeat-compare-and-simulate-longer) | \n| Synthetic webhooks | [Callbacks](https://github.com/firedrill-tools/firedrill/blob/main/docs/callbacks.md) | \n| Reusable starting states | [Scenarios](https://github.com/firedrill-tools/firedrill/blob/main/docs/reusable-scenarios.md) | \n| Logs, screenshots, recordings, and traces | [Capture](https://github.com/firedrill-tools/firedrill/blob/main/docs/capture.md) | \n| UI-driven tests | [Browser tests](https://github.com/firedrill-tools/firedrill/blob/main/packages/browser-tests/README.md) | \n\nBrowser tests drive an application. Tool apps are interfaces to synthetic dependencies. Combine a browser harness with a world-bound drill to check both page behavior and Tool state.\n\nIn CI, use the same commands or SDK tests and retain the report directory. JUnit files work with standard test-results viewers.\n\nFiredrill includes a [skill](https://github.com/firedrill-tools/firedrill/blob/main/skills/firedrill/SKILL.md) for coding agents and an\noptional authoring assistant built with the Claude Agent SDK.\n\nTo install instructions for your coding agent:\n\n```\nnpx firedrill init --path coding-agent\n```\n\nTo use Firedrill Agent:\n\n```\nnpx firedrill init --path firedrill-agent\nnpx firedrill agent\nnpx firedrill agent --workflow drill\n```\n\nThe default workflow creates or edits a synthetic environment. Use\n`--workflow drill` to author tests. Both initialization paths create instructions\nand a repository brief; the agent then authors the definitions.\n\nFiredrill Agent requires `@firedrill-run/agent` and `ANTHROPIC_API_KEY` in the process\nenvironment. For Python, install `python -m pip install \"firedrill-run[agent]\"`.\nThe source checkout includes the package. The CLI does not load\n`.env` automatically. Selected repository content is sent to Anthropic.\n\nDefault limits are 40 turns, $2 of model spend, and 15 minutes per invocation.\nSee `firedrill agent --help` for overrides and [Security](https://github.com/firedrill-tools/firedrill/blob/main/SECURITY.md#optional-firedrill-agent)\nfor execution and source-access boundaries. `--json` diagnostics are available\nfor scripted validation and authoring workflows.\n\nCreate a Tool in your project:\n\n```\nnpx firedrill tool create my-tool\nnpx firedrill tool create my-helper --template stateless\nnpx firedrill tool inspect my-tool\nnpx firedrill tool validate my-tool\n```\n\nTool declarations describe operations and schemas; behavior modules implement\ntheir responses and state changes. Add a conformance suite, then run\n`firedrill tool test my-tool`.\n\nFor a standalone package with a starter conformance suite:\n\n```\nnpx firedrill tool create my-tool --package --name @your-team/my-tool --root <new-directory>\n```\n\nPackages can live in any repository. Install one from npm, Git, a local directory,\nor an archive with `firedrill tool add <source> --install`. Without `--install`,\nthe command selects an already installed package. Use `firedrill tool list` to\nbrowse the community catalog, `firedrill tool search <text>` to narrow it, or\n`--index <file-or-HTTPS-url>` to use another index.\n\nInstallation pins source and disables lifecycle scripts. Tool execution uses your local permissions; review packages as executable test dependencies. Conformance results describe tested coverage, not complete service compatibility.\n\nSee [installation](https://github.com/firedrill-tools/firedrill/blob/main/docs/tool-installation.md), [package authoring](https://github.com/firedrill-tools/firedrill/blob/main/docs/tool-packages.md),\n[Tool apps](https://github.com/firedrill-tools/firedrill/blob/main/docs/tool-apps.md), and the [compatibility contract](https://github.com/firedrill-tools/firedrill/blob/main/docs/tool-compatibility.md).\n\nSee [CONTRIBUTING.md](https://github.com/firedrill-tools/firedrill/blob/main/CONTRIBUTING.md).\n\n[Apache-2.0](https://github.com/firedrill-tools/firedrill/blob/main/LICENSE). Copyright Reload Tech Inc.", "url": "https://wpnews.pro/news/firedrill-stateful-tool-simulation-for-ai-agents", "canonical_source": "https://github.com/firedrill-tools/firedrill", "published_at": "2026-09-22 13:35:34+00:00", "updated_at": "2026-09-22 13:55:01.523131+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols"], "entities": ["Firedrill", "firedrill-run", "@firedrill-run/cli", "@firedrill-run/sdk", "@firedrill-tools", "PyPI", "npm", "Node.js"], "alternates": {"html": "https://wpnews.pro/news/firedrill-stateful-tool-simulation-for-ai-agents", "markdown": "https://wpnews.pro/news/firedrill-stateful-tool-simulation-for-ai-agents.md", "text": "https://wpnews.pro/news/firedrill-stateful-tool-simulation-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/firedrill-stateful-tool-simulation-for-ai-agents.jsonld"}}