Firedrill: Stateful tool simulation for AI agents

Firedrill released a simulation and testing framework for AI agents that lets developers define synthetic stateful tools over HTTP, MCP, CLI, and function bindings and assert on tool calls, state changes, and events. The framework ships as the Python package firedrill-run (Python 3.10+) and the npm packages @firedrill-run/cli and @firedrill-run/sdk, requiring Node.js 20.19 or later, and offers scenario-based tests with faults, response overrides, virtual time, isolated world state, seeded data, snapshots, resets, and HTML, JSON, and JUnit reports. Tools install on demand from a catalog as @firedrill-tools/<id> dev dependencies, with the CLI verifying package integrity before installation consent.

Firedrill is a simulation and testing framework for AI agents. Define synthetic tools and data, run your agent against them, and assert on tool calls, state changes, and events. - Stateful tools with HTTP, MCP, CLI, and function bindings. - Scenario-based tests with faults, response overrides, and virtual time. - Isolated world state, seeded data, snapshots, and resets. - HTML, JSON, and JUnit reports with timelines and optional browser captures. - Repository-defined tools, including independently distributed packages. Quickstart quickstart · Python https://github.com/firedrill-tools/firedrill/blob/main/python/README.md · TypeScript using-the-sdk · Documentation https://docs.firedrill.run · Neutral example https://github.com/firedrill-tools/firedrill/blob/main/examples/quickstart/README.md · Gmail Agent example https://github.com/firedrill-tools/firedrill-example-gmail-agent Install the current release from PyPI https://pypi.org/project/firedrill-run/ into your virtual environment with Python 3.10 or later: python -m pip install "firedrill-run pytest " firedrill --help Import the local SDK with from firedrill import World, run drills . The wheel includes the runtime, CLI, inspector, and report engine. No separate Node.js or npm installation is required. Choose Tools with firedrill init ; they install on demand and use the same packages as TypeScript. See the Python guide https://github.com/firedrill-tools/firedrill/blob/main/python/README.md for pytest, async agents, mocks, and browser tests. Requires Node.js 20.19 or later. Install the current release in your project: npm install --save-dev @firedrill-run/cli @firedrill-run/sdk npx firedrill --help Or run the CLI without adding a project dependency: npx @firedrill-run/cli init For a source checkout, use pnpm 9.15–10: pnpm install --frozen-lockfile pnpm build Use the built CLI in this terminal. export FIREDRILL CLI="$PWD/packages/cli/dist/bin.js" firedrill { node "$FIREDRILL CLI" "$@"; } The examples below use npx firedrill inside a project that has @firedrill-run/cli installed. Never run bare npx firedrill elsewhere: outside such a project npm resolves an unrelated package with that name. Python installations expose firedrill directly, so drop the npx prefix. From source, use the shell function above or invoke the CLI directly with node /path/to/firedrill/packages/cli/dist/bin.js . The programmatic API is @firedrill-run/sdk . To prepare installable archives of the CLI, SDK, and other packages from this checkout, run pnpm pack:artifacts -- --output /absolute/path/to/an/empty/directory . The output includes a package manifest. Create a project with a synthetic record store: mkdir firedrill-example cd firedrill-example npm init -y npm install --save-dev @firedrill-run/cli npx firedrill init --custom records npx firedrill serve init creates a Tool declaration, a behavior module, and starting data. serve starts the backend and opens the inspector. Open Tools to inspect the implementation or call an operation. State & activity shows records and calls; Connect agent provides the connection settings. Tools with a bundled UI also have an Open app action. Browser actions and API calls use the same state. The server listens on loopback using available ports. Keep the terminal open; Ctrl+C stops it. Use --no-open to skip opening the inspector automatically. Run firedrill init in an existing project for guided setup, or select one of the published packages in the Tool catalog https://github.com/firedrill-tools/firedrill/blob/main/registry/README.md , for example npx firedrill init --tool gmail --install or npx firedrill tool add gmail --install . Both add the catalog's exact @firedrill-tools/<id npm release as an ordinary dev dependency and refuse an archive whose integrity differs from the catalog. The CLI shows the exact package and version before it asks for installation consent. Tools can run independently of tests. To check an agent's behavior, add a drill. For a model-backed project, see the Gmail Agent example https://github.com/firedrill-tools/firedrill-example-gmail-agent : an existing Claude Agent SDK assistant runs three drills against a pinned stateful Gmail Tool, with its synthetic mailbox and reports kept in the project. A drill defines an agent task, starting conditions, and assertions about the result. To try one, stop the server, return to the parent directory cd .. , and create the example test project: mkdir firedrill-first-drill cd firedrill-first-drill npm init -y npm install --save-dev @firedrill-run/cli npx firedrill init --path template npx firedrill validate npx firedrill plan npx firedrill run changes-resource npx firedrill inspect The template contains a deterministic example agent that writes 7 to a record. Its drill checks that the write succeeded once and that the final value is 7 . Replace the example target with your agent when adding your own tests. To inspect a failing result, edit firedrill/drills/changes-resource.drill.yaml : change the value-changed assertion's expected value to 8 , keeping task.input.value at 7 . Rerun the drill. The report shows expected 8 and actual 7 ; the process exits with code 1 . Restore the expectation afterwards. | Command | Purpose | |---|---| | firedrill validate | Check source definitions | | firedrill plan | List tools, scenarios, targets, and drills | | firedrill run <id | Run one drill | | firedrill | Run all drills | | firedrill serve | Start a standalone synthetic backend | | firedrill inspect | Browse definitions and saved results | | firedrill ci init github | Add drills to pull requests, pushes, schedules, or manual CI jobs | Run the same repository-owned drill command locally and in CI. A pull request is one useful trigger, not a requirement: pushes, scheduled suites, manual jobs, and other CI providers use the same command and reports. firedrill ci describe firedrill ci init github The initializer detects the project's package manager or Python setup, preserves an existing test:firedrill / drills script when present, and writes .github/workflows/firedrill.yml . It uploads self-contained HTML, JSON, and JUnit reports even when a drill fails. Use --run for a caller-owned SDK harness and repeat --trigger to select exact events. See the continuous-integration guide https://github.com/firedrill-tools/firedrill/blob/main/docs/continuous-integration.md . Add a focused safety suite to pull requests without moving the agent into Firedrill or changing production agent code: firedrill ci init github \ --run "npm run test:firedrill" \ --trigger pull-request \ --trigger manual Commit the generated workflow. GitHub runs the repository's existing command at the exact pull-request revision and shows its exit status as a normal check. Firedrill retains .firedrill/reports/ as an Actions artifact even when a drill fails, so a reviewer can open the HTML result, JUnit output, changed state, and ordered evidence behind the verdict. Requiring that check in branch protection is optional. The hosted workflow adds managed worlds, a base-versus-head behavior comparison, a durable evidence link, and a Firedrill check and pull-request summary. Local CI remains complete and account-free. | Term | Meaning | |---|---| | Tool | A synthetic dependency with callable operations, input/output schemas, and an implementation | | World | Tools, starting data, identities, permissions, and a clock | | Scenario | A variation of the starting data, permissions, faults, or scheduled events | | Target | Configuration for invoking the agent under test | | Drill | A task and its assertions | | Run | A recorded execution result, including checks, calls, and state changes | Actors identify who is calling a tool and which operations they may use. Personas provide descriptions of those identities. See people and permissions https://github.com/firedrill-tools/firedrill/blob/main/docs/world-authoring.md people-and-permissions . Configure the agent's dependencies in test setup: | Dependency | Integration | |---|---| | HTTP client | Point its base URL and authentication at the Tool's declared HTTP routes | | MCP server | Use the world's MCP endpoint and actor token | | CLI tool | Use a test-side command adapter or the Firedrill world CLI | | Function or SDK method | Use mock tool with Python's unittest.mock / pytest, or mockTool with JavaScript runner mocks | | Web interface | Use a Playwright harness or the optional browser-test package | Bindings use existing configuration or test-side adapters, leaving production agent logic unchanged. Hardcoded dependencies need an interceptable boundary or an explicit adapter. See binding recipes https://github.com/firedrill-tools/firedrill/blob/main/docs/quickstart.md 3-keep-the-agent-integration-at-one-seam and test-side mocking https://github.com/firedrill-tools/firedrill/blob/main/docs/test-mocking.md . Targets can invoke a module, start a command, call an HTTP endpoint, or use an external callback supplied by a test harness. External targets run through the SDK; module, command, and HTTP targets can also run through the CLI. Model credentials belong to the agent process. For command targets, pass model credentials and other required host variables through environmentFromHost . Set target timeouts for the complete model/tool loop. Use runDrills from an existing test runner: js import { runDrills } from "@firedrill-run/sdk"; const result = await runDrills { root: process.cwd , drill: "my-drill", agent: { task, binding, signal } = runMyAgent { task, environment: binding.environment, signal } , } ; expect result.verdict .toBe "passed" ; This example assumes a declared my-drill with an external target. runMyAgent is your test adapter; it applies the supplied connection values to your agent. expect comes from your test runner. runDrills { setup } supports per-test data, fault, and Tool overrides. createLocalWorld provides direct control over calls, state, time, and resets. See the SDK reference https://github.com/firedrill-tools/firedrill/blob/main/packages/sdk/README.md for lifecycle hooks, concurrency, capture, and report APIs. The example template uses the following layout: your-project/ firedrill.json source location and selected packages firedrill/ world.yaml starting data, identities, access, time tools/resource-store/ resource-store.tool.yaml operations and state schemas behavior.mjs operation implementations scenarios/baseline.scenario.yaml starting conditions targets/starter-agent.target.yaml agent invocation drills/changes-resource.drill.yaml task and assertions suites/resource-store-conformance.suite.yaml firedrill-example/agent.mjs example agent .firedrill-tools/ vendored Tool dependencies .firedrill/ generated state, builds, and reports Definitions support JSON or YAML; use either consistently or mix them. Resource suffixes identify file types, such as .tool.json and .drill.yaml . References use IDs inside the files, so you can organize folders as needed. Tool implementations are JavaScript or TypeScript. Commit definitions, behavior modules, test code, package manifests, lockfiles, and referenced .firedrill-tools/ archives. Generated files under .firedrill/ are ignored by init . Keep credentials, .env files, and node modules/ ignored as well. Tool state lives in SQLite. Runtime writes leave the source definitions unchanged. Source edits apply to the next build; restart serve to load them. Each trial or retry uses an isolated world. A full reset restores its baseline; a scoped reset restores selected Tools. Reset affects the synthetic environment, not the agent's own database or memory. Saved reports are retained. See source authoring https://github.com/firedrill-tools/firedrill/blob/main/docs/world-authoring.md and world controls https://github.com/firedrill-tools/firedrill/blob/main/docs/local-world-control.md . Open .firedrill/reports/index.html to browse saved runs, or use Results in the inspector. Each run directory contains: - An HTML report with task, outcome, expected/actual checks, and tool activity. - JSON results, JUnit XML, and ordered evidence. - An integrity manifest and any captured attachments. npx firedrill compare .firedrill/reports/<baseline .firedrill/reports/<candidate npx firedrill report verify .firedrill/reports/<run-id Reports distinguish assertion failures, execution errors, and incomplete evidence. CLI exit codes are 0 for a passing selection, 1 for source/execution/check failure, and 2 for invalid command usage. The reproduction command pins the build and seed. This reproduces the world inputs; live model responses can still vary. Comparisons identify changed inputs before presenting result differences. Keep attachment folders with their reports. Review captured data before sharing. Bundle verification checks integrity, not authorship. See running and results https://github.com/firedrill-tools/firedrill/blob/main/docs/running-and-results.md for the report layout and CI use. | Capability | Usage or guide | |---|---| | Repeated trials | npx firedrill run changes-resource --trials 3 --seed 42 | | Filtered, concurrent execution | npx firedrill run --tag smoke --concurrency 4 | | Watch mode | npx firedrill run changes-resource --watch | | Per-test data and Tool overrides | SDK setup https://github.com/firedrill-tools/firedrill/blob/main/packages/sdk/README.md per-test-synthetic-data-and-tools | | State inspection, virtual time, and resets | World controls https://github.com/firedrill-tools/firedrill/blob/main/docs/local-world-control.md | | Multi-interaction simulations | Drill timelines https://github.com/firedrill-tools/firedrill/blob/main/docs/running-and-results.md repeat-compare-and-simulate-longer | | Synthetic webhooks | Callbacks https://github.com/firedrill-tools/firedrill/blob/main/docs/callbacks.md | | Reusable starting states | Scenarios https://github.com/firedrill-tools/firedrill/blob/main/docs/reusable-scenarios.md | | Logs, screenshots, recordings, and traces | Capture https://github.com/firedrill-tools/firedrill/blob/main/docs/capture.md | | UI-driven tests | Browser tests https://github.com/firedrill-tools/firedrill/blob/main/packages/browser-tests/README.md | Browser tests drive an application. Tool apps are interfaces to synthetic dependencies. Combine a browser harness with a world-bound drill to check both page behavior and Tool state. In CI, use the same commands or SDK tests and retain the report directory. JUnit files work with standard test-results viewers. Firedrill includes a skill https://github.com/firedrill-tools/firedrill/blob/main/skills/firedrill/SKILL.md for coding agents and an optional authoring assistant built with the Claude Agent SDK. To install instructions for your coding agent: npx firedrill init --path coding-agent To use Firedrill Agent: npx firedrill init --path firedrill-agent npx firedrill agent npx firedrill agent --workflow drill The default workflow creates or edits a synthetic environment. Use --workflow drill to author tests. Both initialization paths create instructions and a repository brief; the agent then authors the definitions. Firedrill Agent requires @firedrill-run/agent and ANTHROPIC API KEY in the process environment. For Python, install python -m pip install "firedrill-run agent " . The source checkout includes the package. The CLI does not load .env automatically. Selected repository content is sent to Anthropic. Default limits are 40 turns, $2 of model spend, and 15 minutes per invocation. See firedrill agent --help for overrides and Security https://github.com/firedrill-tools/firedrill/blob/main/SECURITY.md optional-firedrill-agent for execution and source-access boundaries. --json diagnostics are available for scripted validation and authoring workflows. Create a Tool in your project: npx firedrill tool create my-tool npx firedrill tool create my-helper --template stateless npx firedrill tool inspect my-tool npx firedrill tool validate my-tool Tool declarations describe operations and schemas; behavior modules implement their responses and state changes. Add a conformance suite, then run firedrill tool test my-tool . For a standalone package with a starter conformance suite: npx firedrill tool create my-tool --package --name @your-team/my-tool --root <new-directory Packages can live in any repository. Install one from npm, Git, a local directory, or an archive with firedrill tool add <source --install . Without --install , the command selects an already installed package. Use firedrill tool list to browse the community catalog, firedrill tool search <text to narrow it, or --index <file-or-HTTPS-url to use another index. Installation pins source and disables lifecycle scripts. Tool execution uses your local permissions; review packages as executable test dependencies. Conformance results describe tested coverage, not complete service compatibility. See installation https://github.com/firedrill-tools/firedrill/blob/main/docs/tool-installation.md , package authoring https://github.com/firedrill-tools/firedrill/blob/main/docs/tool-packages.md , Tool apps https://github.com/firedrill-tools/firedrill/blob/main/docs/tool-apps.md , and the compatibility contract https://github.com/firedrill-tools/firedrill/blob/main/docs/tool-compatibility.md . See CONTRIBUTING.md https://github.com/firedrill-tools/firedrill/blob/main/CONTRIBUTING.md . Apache-2.0 https://github.com/firedrill-tools/firedrill/blob/main/LICENSE . Copyright Reload Tech Inc.