Testing Agent Experience in Agent-Native Dev Tools

Shen Li released DevTool AX Kit, a local-first toolkit for testing agent-native developer tools, which captures agent runs, verifies outcomes with independent checks, and safely replays external effects without repeating writes. The toolkit, requiring Python 3.11 or newer, measures whether coding agents can find instructions, recover from ambiguous responses, and have their success claims independently verified.

Measure whether developer tools are understandable, recoverable, and verifiable for coding agents. DevTool AX Kit is a small, local-first toolkit for testing Agent-Native developer tools. It captures what happened, verifies what actually happened, and makes ambiguous external effects safe to inspect without repeating them. Agent success is more than “the model produced code.” A useful evaluation asks: - Could the agent find the right instructions and interfaces? - Could it recover after an ambiguous tool response? - Did an independent verifier agree with its success claim? - Were retries, credentials, and external side effects contained? | Capability | Use it to measure | |---|---| | Run capture and snapshots | What the agent saw and changed | | Checkpoints and pinned replay | Whether uncertain work can be resumed safely | | Effect receipts | Whether an external write is identified and recoverable | | Redaction and safety assertions | Whether evidence can be shared safely | | AX skills and templates | How to design, verify, compare, and report tasks | Requires Python 3.11 or newer. No runtime dependencies are required. git clone https://github.com/shenli/devtool-ax-kit.git cd devtool-ax-kit python3 -m unittest discover -s tests -v python3 scripts/replay experiment.py python3 -m agent run --help Capture a local command: python3 -m agent run capture --workspace /path/to/workspace -- python3 -m unittest Record a receipted effect, then replay it without executing the write: python3 -m agent run record-tool --run RUN ID \ --name example.send --input '{"event":"welcome/42"}' \ --output '{"status":"accepted"}' --effect write \ --receipt '{"request id":"demo-123"}' --authoritative-status accepted python3 -m agent run checkpoint --run RUN ID --workspace /path/to/workspace --label after-write python3 -m agent run replay --checkpoint CHECKPOINT ID Pinned replay returns the original receipt without executing the write; an unrecorded write is rejected. agent run/ capture, snapshots, checkpoints, replay skills/ reusable AX evaluation instructions docs/ methodology, taxonomy, and report templates tests/ local regression tests scripts/replay experiment.py synthetic mechanics demonstration Start with AGENTS.md /shenli/devtool-ax-kit/blob/main/AGENTS.md , then choose the relevant skill under . The /shenli/devtool-ax-kit/blob/main/skills skills/ documentation index /shenli/devtool-ax-kit/blob/main/docs/README.md links the methodology and example artifacts. This is evaluation instrumentation—not a credential vault, sandbox provider, orchestrator, security audit, or exact model-replay system. Keep experiments local or use disposable test resources, and run independent verification outside the agent's editable workspace. Keep tasks vendor-neutral, verifiers deterministic, and evidence reproducible. Run the unit tests and skill validators before opening a pull request. See SECURITY.md /shenli/devtool-ax-kit/blob/main/SECURITY.md before sharing trajectories or external-run data.