{"slug": "multi-turn-memory-and-multi-agent-evaluation", "title": "Multi-Turn, Memory, and Multi-Agent Evaluation", "summary": "A tutorial on multi-turn, memory, and multi-agent evaluation argues that single-turn LLM evals miss conversational failures such as re-asking for information already given, forgetting user constraints, over-asking and under-asking clarifying questions, and state drift between agent and user. The lesson's core technique is an LLM-driven user simulator configured with a goal, persona, hidden facts revealed only when asked, and constraints, plus [DONE] and [GIVE_UP] termination tokens and a max_turns cap of 8 in the sample UserScenario. It also covers conversation-level success and turn-level metrics, memory evaluations for recall, update and forgetting, and attributing failures to specific agents and handoffs in multi-agent systems.", "body_md": "· Agentic AI · 7 min read\n\n### 📋 Prerequisites\n\n- Trajectory and Tool-Use Evaluation (previous lesson)\n\n### 🎯 What You'll Learn\n\n- Build an LLM-driven user simulator with goals, personas, and hidden information\n- Define conversation-level success and turn-level metrics for multi-turn agents\n- Design memory evaluations for recall, update, and forgetting\n- Attribute failures to specific agents and handoffs in a multi-agent system\n\n## Single-Turn Evals Hide Conversational Failures\n\nEverything so far assumed one user message per case. Real support conversations aren’t like that. The customer leaves out their email, the agent asks, the customer gives the wrong one, then corrects it. The customer changes their mind halfway through. The customer gets frustrated after the second clarifying question.\n\nFailures that only show up across turns:\n\n- **Asking for information already given** , two turns earlier\n- **Forgetting a constraint** (“don’t escalate me, I’ve been waiting three days”)\n- **Over-asking** : four clarifying questions when one would do\n- **Under-asking** : guessing the account when the email was ambiguous\n- **State drift** : the agent’s understanding of the problem diverges from the user’s\n\nTo evaluate these, the test needs a counterpart that talks back.\n\n## The User Simulator\n\nA user simulator is an LLM playing the customer. It gets a **goal**, a **persona**, and **hidden information** it reveals only when asked. That last part is critical. If the simulator states everything upfront, you’re back to a single-turn eval.\n\n``` python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass UserScenario:\n    goal: str                      # what the user wants, in their terms\n    persona: str                   # tone, patience, expertise\n    opening: str                   # first message\n    hidden_facts: dict[str, str]   # revealed only if the agent asks\n    constraints: list[str] = field(default_factory=list)\n    max_turns: int = 8\n\nSIMULATOR_PROMPT = \"\"\"You are role-playing a customer contacting billing support.\n\nGoal: {goal}\nPersona: {persona}\nFacts you know (reveal a fact ONLY if the agent asks for it or it is clearly relevant):\n{facts}\nYour constraints: {constraints}\n\nRules:\n- Stay in character. Write short, realistic messages.\n- Never volunteer facts the agent has not asked about.\n- If your goal is fully achieved, reply with exactly: [DONE]\n- If the agent has clearly failed or is going in circles, reply with exactly: [GIVE_UP]\n\"\"\"\n\nclass UserSimulator:\n    def __init__(self, scenario: UserScenario, llm):\n        self.s, self.llm = scenario, llm\n        facts = \"\\n\".join(f\"- {k}: {v}\" for k, v in scenario.hidden_facts.items())\n        self.system = SIMULATOR_PROMPT.format(goal=scenario.goal, persona=scenario.persona,\n                                              facts=facts,\n                                              constraints=\"; \".join(scenario.constraints) or \"none\")\n\n    def reply(self, transcript: list[dict]) -> str:\n        # Flip roles: the agent's messages are the 'user' from the simulator's view\n        flipped = [{\"role\": \"user\" if m[\"role\"] == \"assistant\" else \"assistant\",\n                    \"content\": m[\"content\"]} for m in transcript]\n        return self.llm(system=self.system, messages=flipped)\n```\n\nRunning a conversation:\n\n``` php\ndef run_conversation(agent, sim: UserSimulator, state) -> dict:\n    transcript = [{\"role\": \"user\", \"content\": sim.s.opening}]\n    for turn in range(sim.s.max_turns):\n        agent_msg = agent.respond(transcript, state)     # may call tools internally\n        transcript.append({\"role\": \"assistant\", \"content\": agent_msg})\n        user_msg = sim.reply(transcript)\n        if user_msg.strip() in (\"[DONE]\", \"[GIVE_UP]\"):\n            return {\"transcript\": transcript, \"end\": user_msg.strip(),\n                    \"turns\": turn + 1, \"final_state\": state.snapshot()}\n        transcript.append({\"role\": \"user\", \"content\": user_msg})\n    return {\"transcript\": transcript, \"end\": \"max_turns\",\n            \"turns\": sim.s.max_turns, \"final_state\": state.snapshot()}\n```\n\nAn example scenario:\n\n```\nUserScenario(\n    goal=\"Get refunded for a duplicate $49 March charge\",\n    persona=\"Frustrated, terse, has already contacted support once\",\n    opening=\"charged twice again. fix it\",\n    hidden_facts={\n        \"email\": \"dana@example.com\",\n        \"month\": \"March\",\n        \"previous_ticket\": \"Contacted support 3 days ago, no answer\",\n    },\n    constraints=[\"Gets annoyed if asked more than two questions\",\n                 \"Does not want to be transferred to a human\"],\n)\n```\n\n### Simulator Pitfalls\n\n**The simulator is part of the test.** If it’s too cooperative, the agent looks better than it is. If it’s inconsistent (forgets its own facts, changes its goal), you get noisy, unfair failures. Validate it:\n\n- Read 20 simulated conversations by hand before trusting the setup\n- Check that the simulator never reveals hidden facts unprompted (a simple grader can check this)\n- Run each scenario several times. Simulator variance stacks on top of agent variance, so you need more trials than for single-turn cases (Lesson 10).\n\n**Don’t let the simulator decide success.** `[DONE]` is a useful stopping signal, but the simulator can be fooled by a confident agent just like a real user. Grade success from **state** and a separate judge, not from the simulator’s own verdict.\n\n## Conversation-Level Metrics\n\n| Metric | How to measure | \n|---|---|\n| **Goal success** | State assertion (refund exists) plus judge on the final resolution | \n| **Turns to resolution** | Count turns until success; compare to a reference | \n| **Clarification efficiency** | Number of agent questions vs number of hidden facts actually needed | \n| **Redundant asks** | Agent asked for a fact already provided earlier in the conversation | \n| **Constraint adherence** | Judge checks each scenario constraint against the transcript | \n| **Abandonment rate** | Fraction ending in `[GIVE_UP]` or max turns | \n\nRedundant asks are checkable without a judge if you tag facts:\n\n``` php\ndef redundant_asks(transcript: list[dict], fact_patterns: dict[str, str]) -> list[str]:\n    \"\"\"fact_patterns: fact name -> regex that detects the agent asking for it.\"\"\"\n    import re\n    provided, redundant = set(), []\n    for m in transcript:\n        if m[\"role\"] == \"user\":\n            for fact, _ in fact_patterns.items():\n                if fact_in_message(fact, m[\"content\"]):\n                    provided.add(fact)\n        else:\n            for fact, pattern in fact_patterns.items():\n                if fact in provided and re.search(pattern, m[\"content\"], re.I):\n                    redundant.append(fact)\n    return redundant\n```\n\n## Evaluating Memory\n\nAgents with memory across sessions (user preferences, prior tickets, account notes) need their own tests. Three properties matter:\n\n**Recall.** Information stored in session 1 is used correctly in session 2.\n\n**Update.** When information changes (“I moved to the annual plan”), the agent uses the new value, not the old one.\n\n**Forgetting and isolation.** Information that should be deleted is gone; one user’s memory never appears in another user’s session.\n\nMemory tests are multi-session scenarios:\n\n```\nmemory_case = {\n    \"sessions\": [\n        {\"user\": \"Please always email receipts to billing@acme.example, not my personal address.\"},\n        {\"user\": \"Can you resend my April receipt?\",\n         \"expect_tool_args\": {\"send_receipt\": {\"to\": \"billing@acme.example\"}}},\n        {\"user\": \"Actually, send receipts to finance@acme.example from now on.\"},\n        {\"user\": \"Resend May's receipt please.\",\n         \"expect_tool_args\": {\"send_receipt\": {\"to\": \"finance@acme.example\"}}},\n    ]\n}\n```\n\n**Isolation tests are safety tests.** Seed memory for user A, run a session as user B, and assert that none of A’s data appears in B’s tool calls or responses. This belongs in the adversarial suite (Lesson 9) as a gate.\n\nFor long single conversations, also test **context-window pressure**: pad the conversation with many turns, then check whether an early constraint is still honored. Agents that summarize or truncate history often lose exactly the constraint that mattered.\n\n## Evaluating Multi-Agent Systems\n\nIn a supervisor/worker design, a request flows through several agents. Imagine Quill v2:\n\n```\n             ┌─────────────┐\n user ─────▶ │ Supervisor  │ ─────▶ response\n             └──┬───────┬──┘\n        route   │       │   route\n          ┌─────▼──┐ ┌──▼──────────┐\n          │Billing │ │ Policy/KB   │\n          │ worker │ │   worker    │\n          └────────┘ └─────────────┘\n```\n\nA wrong final answer could come from a bad routing decision, a worker error, a lossy handoff, or the supervisor misreading a worker’s result. End-to-end evaluation tells you *that* it failed. You need **attribution** to know *where*.\n\n### Evaluate Each Role, Then the System\n\n1. **Routing accuracy.** Step cases for the supervisor: given this request, which worker should handle it?\n2. **Worker quality.** Each worker evaluated in isolation with its own dataset, as a single agent.\n3. **Handoff fidelity.** Does the message the supervisor sends a worker contain everything the worker needs? A judge comparing the original request to the handoff message works well.\n4. **Synthesis.** Given correct worker outputs, does the supervisor compose a correct final response?\n5. **End to end.** The full system on the golden dataset.\n\n### Attribution With Traces\n\nBecause each agent’s activity is a subtree of spans (Lesson 2), you can attribute failures automatically: walk the trace and run the per-role graders on each subtree.\n\n``` php\ndef attribute_failure(trace, case) -> str:\n    route = first_span(trace, kind=\"agent\", name=\"supervisor.route\")\n    if route.output[\"worker\"] not in case.expected[\"acceptable_workers\"]:\n        return \"routing\"\n    worker_root = first_span(trace, kind=\"agent\", name=route.output[\"worker\"])\n    if not worker_graders_pass(worker_root, case):\n        return f\"worker:{route.output['worker']}\"\n    if not handoff_complete(route.input, worker_root.input):\n        return \"handoff\"\n    return \"synthesis\"\n```\n\nReport failures by attribution bucket:\n\n```\nend-to-end pass rate   81%\nfailures by stage:\n  routing              31%\n  worker:billing       22%\n  handoff              38%\n  synthesis             9%\n```\n\nA breakdown like that changes what you work on next. Here, improving handoff messages is worth more than improving the billing worker.\n\n### Multi-Agent Specific Failure Modes\n\n- **Ping-pong** : agents hand a task back and forth without progress. Grade with a max-handoffs budget.\n- **Duplicated work** : two workers both perform the same action (two refunds!). The no-collateral-changes and cardinality gates from earlier lessons catch this.\n- **Authority confusion** : a worker takes an action only the supervisor should approve. Check permissions per agent in the trace.\n\n## Key Takeaways\n\n- Many agent failures only appear across turns; single-turn evals can’t see them\n- A **user simulator** needs a goal, persona, and**hidden facts** revealed only when asked\n- Validate the simulator, and never let it be the judge of success; grade from state and a separate judge\n- Track turns to resolution, clarification efficiency, redundant asks, constraint adherence, and abandonment\n- Memory evals test **recall** ,**update** , and**isolation** ; isolation is a safety gate\n- In multi-agent systems, evaluate each role, then **attribute failures** using trace subtrees\n\n## Exercise\n\n1. Build a user simulator and write 10 scenarios with hidden facts and constraints for your agent\n2. Run each scenario 5 times; report goal success, mean turns, redundant asks, and abandonment\n3. Read five transcripts and note any simulator misbehavior; fix the simulator prompt\n4. If you have a multi-agent system, implement `attribute_failure` and report failures by stage. If not, write a two-session memory test with an update step.", "url": "https://wpnews.pro/news/multi-turn-memory-and-multi-agent-evaluation", "canonical_source": "https://superml.org/tutorials/agent-eval-multi-turn-multi-agent", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-19 03:43:22.389824+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools"], "entities": [], "also_reported_by": [], "takeaway": "Give the simulated user hidden facts it reveals only when the agent asks, and cap conversations at max_turns (8 in the sample) with explicit [DONE] and [GIVE_UP] tokens, so multi-turn evals surface over-asking, forgotten constraints and state drift that single-turn cases hide.", "recommendation": {"advice": "Build multi-turn evals with an LLM user simulator that holds hidden facts and reveals them only when asked, and end runs on [DONE], [GIVE_UP] or a max_turns cap to catch over-asking and state drift.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/multi-turn-memory-and-multi-agent-evaluation", "markdown": "https://wpnews.pro/news/multi-turn-memory-and-multi-agent-evaluation.md", "text": "https://wpnews.pro/news/multi-turn-memory-and-multi-agent-evaluation.txt", "jsonld": "https://wpnews.pro/news/multi-turn-memory-and-multi-agent-evaluation.jsonld"}}