Show HN: Understudy: Scenario Testing for AI Agents Goji Plus released Understudy, an open-source scenario-driven testing framework for AI agents that simulates multi-turn users, records execution traces, and asserts on tool calls rather than prose. The framework supports ADK, LangGraph, and HTTP agents, offers mock toolkits, YAML scene definitions, and pytest integration, and is installable via `pip install understudy[all]`. Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports. Testing with understudy is 4 steps : Wrap your agent — Adapt your agent ADK, LangGraph, HTTP to understudy's interface Mock your tools — Register handlers that return test data instead of calling real services Write scenes — YAML files defining what the simulated user wants and what you expect Run and assert — Execute simulations, check traces, generate reports The key insight: assert against the trace, not the prose . Don't check what the agent said—check what it did tool calls . Simulate multi-turn conversations with personas to test dialogue agents. - Use case: Customer service bots, assistants, chatbots - Assert on tool calls: trace.called "tool name" Evaluate autonomous agents executing multi-step tasks. - Use case: Code agents, research agents, task automation - Assert on actions: trace.performed "action" See examples/README.md /gojiplus/understudy/blob/main/examples/README.md for complete examples of both paradigms. See real examples: Example scene https://github.com/gojiplus/understudy/blob/main/examples/scenes/return eligible backpack.yaml — YAML defining a test scenario ADK test file https://github.com/gojiplus/understudy/blob/main/examples/adk/test adk returns.py — pytest assertions against traces LangGraph test file https://github.com/gojiplus/understudy/blob/main/examples/langgraph/test langgraph returns.py — same tests, different framework Agentic test file https://github.com/gojiplus/understudy/blob/main/examples/agentic/test agentic.py — agentic flow evaluation Agentic scene https://github.com/gojiplus/understudy/blob/main/examples/agentic scenes/code review task.yaml — task-based scenario Example report https://htmlpreview.github.io/?https://github.com/gojiplus/understudy/blob/main/examples/langgraph/report/index.html — HTML report with metrics and transcripts pip install understudy all python from understudy.adk import ADKApp from my agent import agent app = ADKApp agent=agent Your agent has tools that call external services. Mock them for testing: python from understudy.mocks import MockToolkit mocks = MockToolkit @mocks.handle "lookup order" def lookup order order id: str - dict: return {"order id": order id, "items": ... , "status": "delivered"} @mocks.handle "create return" def create return order id: str, item sku: str, reason: str - dict: return {"return id": "RET-001", "status": "created"} Create scenes/return backpack.yaml : id: return eligible backpack description: Customer wants to return a backpack starting prompt: "I'd like to return an item please." conversation plan: | Goal: Return the hiking backpack from order ORD-10031. - Provide order ID when asked - Return reason: too small persona: cooperative max turns: 15 expectations: required tools: - lookup order - create return forbidden tools: - issue refund python from understudy import Scene, run scene = Scene.from file "scenes/return backpack.yaml" trace = run app, scene, mocks=mocks assert trace.called "lookup order" assert trace.called "create return" assert not trace.called "issue refund" Or with pytest define app and mocks fixtures in conftest.py : pytest test returns.py -v Run multiple scenes with multiple simulations per scene: python from understudy import Suite, RunStorage suite = Suite.from directory "scenes/" storage = RunStorage Run each scene 3 times and tag for comparison results = suite.run app, mocks=mocks, storage=storage, n sims=3, tags={"version": "v1"}, print f"{results.pass count}/{len results.results } passed" Understudy separates simulation generating traces from evaluation checking traces . Use together or separately: understudy run \ --app mymodule:agent app \ --scene ./scenes/ \ --n-sims 3 \ --junit results.xml Generate traces only: understudy simulate \ --app mymodule:agent app \ --scenes ./scenes/ \ --output ./traces/ \ --n-sims 3 Evaluate existing traces: understudy evaluate \ --traces ./traces/ \ --output ./results/ \ --junit results.xml Python API: python from understudy import simulate batch, evaluate batch Generate traces traces = simulate batch app=agent app, scenes="./scenes/", n sims=3, output="./traces/", Evaluate later results = evaluate batch traces="./traces/", output="./results/", Run simulations understudy run --app mymodule:app --scene ./scenes/ understudy simulate --app mymodule:app --scenes ./scenes/ understudy evaluate --traces ./traces/ View results understudy list understudy show