{"slug": "show-hn-understudy-scenario-testing-for-ai-agents", "title": "Show HN: Understudy: Scenario Testing for AI Agents", "summary": "Goji Plus released Understudy, an open-source scenario-driven testing framework for AI agents that simulates multi-turn users, records execution traces, and asserts on tool calls rather than prose. The framework supports ADK, LangGraph, and HTTP agents, offers mock toolkits, YAML scene definitions, and pytest integration, and is installable via `pip install understudy[all]`.", "body_md": "Understudy is a scenario-driven testing framework for AI agents that simulates realistic multi-turn users, runs those scenes against an agent through a simple app adapter, records a structured execution trace of messages, tool calls, and handoffs, and then evaluates behavior with deterministic checks, optional LLM judges, and run reports.\n\nTesting with understudy is **4 steps**:\n\n**Wrap your agent**— Adapt your agent (ADK, LangGraph, HTTP) to understudy's interface** Mock your tools**— Register handlers that return test data instead of calling real services** Write scenes**— YAML files defining what the simulated user wants and what you expect** Run and assert**— Execute simulations, check traces, generate reports\n\nThe key insight: **assert against the trace, not the prose**. Don't check what the agent said—check what it did (tool calls).\n\nSimulate multi-turn conversations with personas to test dialogue agents.\n\n- Use case: Customer service bots, assistants, chatbots\n- Assert on tool calls:\n`trace.called(\"tool_name\")`\n\nEvaluate autonomous agents executing multi-step tasks.\n\n- Use case: Code agents, research agents, task automation\n- Assert on actions:\n`trace.performed(\"action\")`\n\nSee [examples/README.md](/gojiplus/understudy/blob/main/examples/README.md) for complete examples of both paradigms.\n\n**See real examples:**\n\n[Example scene](https://github.com/gojiplus/understudy/blob/main/examples/scenes/return_eligible_backpack.yaml)— YAML defining a test scenario[ADK test file](https://github.com/gojiplus/understudy/blob/main/examples/adk/test_adk_returns.py)— pytest assertions against traces[LangGraph test file](https://github.com/gojiplus/understudy/blob/main/examples/langgraph/test_langgraph_returns.py)— same tests, different framework[Agentic test file](https://github.com/gojiplus/understudy/blob/main/examples/agentic/test_agentic.py)— agentic flow evaluation[Agentic scene](https://github.com/gojiplus/understudy/blob/main/examples/agentic_scenes/code_review_task.yaml)— task-based scenario[Example report](https://htmlpreview.github.io/?https://github.com/gojiplus/understudy/blob/main/examples/langgraph/report/index.html)— HTML report with metrics and transcripts\n\n```\npip install understudy[all]\npython\nfrom understudy.adk import ADKApp\nfrom my_agent import agent\n\napp = ADKApp(agent=agent)\n```\n\nYour agent has tools that call external services. Mock them for testing:\n\n``` python\nfrom understudy.mocks import MockToolkit\n\nmocks = MockToolkit()\n\n@mocks.handle(\"lookup_order\")\ndef lookup_order(order_id: str) -> dict:\n    return {\"order_id\": order_id, \"items\": [...], \"status\": \"delivered\"}\n\n@mocks.handle(\"create_return\")\ndef create_return(order_id: str, item_sku: str, reason: str) -> dict:\n    return {\"return_id\": \"RET-001\", \"status\": \"created\"}\n```\n\nCreate `scenes/return_backpack.yaml`\n\n:\n\n```\nid: return_eligible_backpack\ndescription: Customer wants to return a backpack\n\nstarting_prompt: \"I'd like to return an item please.\"\nconversation_plan: |\n  Goal: Return the hiking backpack from order ORD-10031.\n  - Provide order ID when asked\n  - Return reason: too small\n\npersona: cooperative\nmax_turns: 15\n\nexpectations:\n  required_tools:\n    - lookup_order\n    - create_return\n  forbidden_tools:\n    - issue_refund\npython\nfrom understudy import Scene, run\n\nscene = Scene.from_file(\"scenes/return_backpack.yaml\")\ntrace = run(app, scene, mocks=mocks)\n\nassert trace.called(\"lookup_order\")\nassert trace.called(\"create_return\")\nassert not trace.called(\"issue_refund\")\n```\n\nOr with pytest (define `app`\n\nand `mocks`\n\nfixtures in conftest.py):\n\n```\npytest test_returns.py -v\n```\n\nRun multiple scenes with multiple simulations per scene:\n\n``` python\nfrom understudy import Suite, RunStorage\n\nsuite = Suite.from_directory(\"scenes/\")\nstorage = RunStorage()\n\n# Run each scene 3 times and tag for comparison\nresults = suite.run(\n    app,\n    mocks=mocks,\n    storage=storage,\n    n_sims=3,\n    tags={\"version\": \"v1\"},\n)\nprint(f\"{results.pass_count}/{len(results.results)} passed\")\n```\n\nUnderstudy separates simulation (generating traces) from evaluation (checking traces). Use together or separately:\n\n```\nunderstudy run \\\n  --app mymodule:agent_app \\\n  --scene ./scenes/ \\\n  --n-sims 3 \\\n  --junit results.xml\n```\n\nGenerate traces only:\n\n```\nunderstudy simulate \\\n  --app mymodule:agent_app \\\n  --scenes ./scenes/ \\\n  --output ./traces/ \\\n  --n-sims 3\n```\n\nEvaluate existing traces:\n\n```\nunderstudy evaluate \\\n  --traces ./traces/ \\\n  --output ./results/ \\\n  --junit results.xml\n```\n\nPython API:\n\n``` python\nfrom understudy import simulate_batch, evaluate_batch\n\n# Generate traces\ntraces = simulate_batch(\n    app=agent_app,\n    scenes=\"./scenes/\",\n    n_sims=3,\n    output=\"./traces/\",\n)\n\n# Evaluate later\nresults = evaluate_batch(\n    traces=\"./traces/\",\n    output=\"./results/\",\n)\n# Run simulations\nunderstudy run --app mymodule:app --scene ./scenes/\nunderstudy simulate --app mymodule:app --scenes ./scenes/\nunderstudy evaluate --traces ./traces/\n\n# View results\nunderstudy list\nunderstudy show <run_id>\nunderstudy summary\n\n# Compare runs by tag\nunderstudy compare --tag version --before v1 --after v2\n\n# Generate reports\nunderstudy report -o report.html\nunderstudy compare --tag version --before v1 --after v2 --html comparison.html\n\n# Interactive browser\nunderstudy serve --port 8080\n\n# HTTP simulator server (for browser/UI testing)\nunderstudy serve-api --port 8000\n\n# Cleanup\nunderstudy delete <run_id>\nunderstudy clear\n```\n\nFor qualities that can't be checked deterministically:\n\n``` python\nfrom understudy.judges import Judge\n\nempathy_judge = Judge(\n    rubric=\"The agent acknowledged frustration and was empathetic while enforcing policy.\",\n    samples=5,\n)\n\nresult = empathy_judge.evaluate(trace)\nassert result.score == 1\n```\n\nBuilt-in rubrics:\n\n```\nfrom understudy.judges import (\n    TOOL_USAGE_CORRECTNESS,\n    POLICY_COMPLIANCE,\n    TONE_EMPATHY,\n    ADVERSARIAL_ROBUSTNESS,\n    TASK_COMPLETION,\n)\n```\n\nThe `understudy summary`\n\ncommand shows:\n\n**Pass rate**— percentage of scenes that passed all expectations** Avg turns**— average conversation length** Tool usage**— distribution of tool calls across runs** Agents**— which agents were invoked\n\nThe HTML report (`understudy report`\n\n) includes:\n\n- All metrics above\n- Full conversation transcripts\n- Tool call details with arguments\n- Expectation check results\n- Judge evaluation results (when used)\n\nSee the [full documentation](https://gojiplus.github.io/understudy) for:\n\n[Installation guide](https://gojiplus.github.io/understudy/installation.html)[Writing scenes](https://gojiplus.github.io/understudy/tutorial/scenes.html)[ADK integration](https://gojiplus.github.io/understudy/adk-integration.html)[LangGraph integration](https://gojiplus.github.io/understudy/langgraph-integration.html)[HTTP client for deployed agents](https://gojiplus.github.io/understudy/tutorial/http.html)[API reference](https://gojiplus.github.io/understudy/api/index.html)\n\nMIT", "url": "https://wpnews.pro/news/show-hn-understudy-scenario-testing-for-ai-agents", "canonical_source": "https://github.com/gojiplus/understudy", "published_at": "2026-08-28 03:51:56+00:00", "updated_at": "2026-08-28 04:18:19.401686+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["Goji Plus", "Understudy", "ADK", "LangGraph"], "alternates": {"html": "https://wpnews.pro/news/show-hn-understudy-scenario-testing-for-ai-agents", "markdown": "https://wpnews.pro/news/show-hn-understudy-scenario-testing-for-ai-agents.md", "text": "https://wpnews.pro/news/show-hn-understudy-scenario-testing-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/show-hn-understudy-scenario-testing-for-ai-agents.jsonld"}}