{"slug": "testing-agent-experience-in-agent-native-dev-tools", "title": "Testing Agent Experience in Agent-Native Dev Tools", "summary": "Shen Li released DevTool AX Kit, a local-first toolkit for testing agent-native developer tools, which captures agent runs, verifies outcomes with independent checks, and safely replays external effects without repeating writes. The toolkit, requiring Python 3.11 or newer, measures whether coding agents can find instructions, recover from ambiguous responses, and have their success claims independently verified.", "body_md": "**Measure whether developer tools are understandable, recoverable, and\nverifiable for coding agents.**\n\nDevTool AX Kit is a small, local-first toolkit for testing Agent-Native developer tools. It captures what happened, verifies what actually happened, and makes ambiguous external effects safe to inspect without repeating them.\n\nAgent success is more than “the model produced code.” A useful evaluation asks:\n\n- Could the agent find the right instructions and interfaces?\n- Could it recover after an ambiguous tool response?\n- Did an independent verifier agree with its success claim?\n- Were retries, credentials, and external side effects contained?\n\n| Capability | Use it to measure |\n|---|---|\n| Run capture and snapshots | What the agent saw and changed |\n| Checkpoints and pinned replay | Whether uncertain work can be resumed safely |\n| Effect receipts | Whether an external write is identified and recoverable |\n| Redaction and safety assertions | Whether evidence can be shared safely |\n| AX skills and templates | How to design, verify, compare, and report tasks |\n\nRequires Python 3.11 or newer. No runtime dependencies are required.\n\n```\ngit clone https://github.com/shenli/devtool-ax-kit.git\ncd devtool-ax-kit\npython3 -m unittest discover -s tests -v\npython3 scripts/replay_experiment.py\npython3 -m agent_run --help\n```\n\nCapture a local command:\n\n```\npython3 -m agent_run capture --workspace /path/to/workspace -- python3 -m unittest\n```\n\nRecord a receipted effect, then replay it without executing the write:\n\n```\npython3 -m agent_run record-tool --run RUN_ID \\\n  --name example.send --input '{\"event\":\"welcome/42\"}' \\\n  --output '{\"status\":\"accepted\"}' --effect write \\\n  --receipt '{\"request_id\":\"demo-123\"}' --authoritative-status accepted\npython3 -m agent_run checkpoint --run RUN_ID --workspace /path/to/workspace --label after-write\npython3 -m agent_run replay --checkpoint CHECKPOINT_ID\n```\n\nPinned replay returns the original receipt without executing the write; an unrecorded write is rejected.\n\n```\nagent_run/                  capture, snapshots, checkpoints, replay\nskills/                     reusable AX evaluation instructions\ndocs/                       methodology, taxonomy, and report templates\ntests/                      local regression tests\nscripts/replay_experiment.py synthetic mechanics demonstration\n```\n\nStart with [ AGENTS.md](/shenli/devtool-ax-kit/blob/main/AGENTS.md), then choose the relevant skill under\n\n[. The](/shenli/devtool-ax-kit/blob/main/skills)\n\n`skills/`\n\n[documentation index](/shenli/devtool-ax-kit/blob/main/docs/README.md)links the methodology and example artifacts.\n\nThis is evaluation instrumentation—not a credential vault, sandbox provider, orchestrator, security audit, or exact model-replay system. Keep experiments local or use disposable test resources, and run independent verification outside the agent's editable workspace.\n\nKeep tasks vendor-neutral, verifiers deterministic, and evidence reproducible.\nRun the unit tests and skill validators before opening a pull request. See\n[ SECURITY.md](/shenli/devtool-ax-kit/blob/main/SECURITY.md) before sharing trajectories or external-run data.", "url": "https://wpnews.pro/news/testing-agent-experience-in-agent-native-dev-tools", "canonical_source": "https://github.com/shenli/devtool-ax-kit", "published_at": "2026-08-28 16:25:02+00:00", "updated_at": "2026-08-28 16:48:20.276778+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools"], "entities": ["Shen Li", "DevTool AX Kit"], "alternates": {"html": "https://wpnews.pro/news/testing-agent-experience-in-agent-native-dev-tools", "markdown": "https://wpnews.pro/news/testing-agent-experience-in-agent-native-dev-tools.md", "text": "https://wpnews.pro/news/testing-agent-experience-in-agent-native-dev-tools.txt", "jsonld": "https://wpnews.pro/news/testing-agent-experience-in-agent-native-dev-tools.jsonld"}}