Multi-Turn, Memory, and Multi-Agent Evaluation A tutorial on multi-turn, memory, and multi-agent evaluation argues that single-turn LLM evals miss conversational failures such as re-asking for information already given, forgetting user constraints, over-asking and under-asking clarifying questions, and state drift between agent and user. The lesson's core technique is an LLM-driven user simulator configured with a goal, persona, hidden facts revealed only when asked, and constraints, plus [DONE] and [GIVE_UP] termination tokens and a max_turns cap of 8 in the sample UserScenario. It also covers conversation-level success and turn-level metrics, memory evaluations for recall, update and forgetting, and attributing failures to specific agents and handoffs in multi-agent systems. Β· Agentic AI Β· 7 min read πŸ“‹ Prerequisites - Trajectory and Tool-Use Evaluation previous lesson 🎯 What You'll Learn - Build an LLM-driven user simulator with goals, personas, and hidden information - Define conversation-level success and turn-level metrics for multi-turn agents - Design memory evaluations for recall, update, and forgetting - Attribute failures to specific agents and handoffs in a multi-agent system Single-Turn Evals Hide Conversational Failures Everything so far assumed one user message per case. Real support conversations aren’t like that. The customer leaves out their email, the agent asks, the customer gives the wrong one, then corrects it. The customer changes their mind halfway through. The customer gets frustrated after the second clarifying question. Failures that only show up across turns: - Asking for information already given , two turns earlier - Forgetting a constraint β€œdon’t escalate me, I’ve been waiting three days” - Over-asking : four clarifying questions when one would do - Under-asking : guessing the account when the email was ambiguous - State drift : the agent’s understanding of the problem diverges from the user’s To evaluate these, the test needs a counterpart that talks back. The User Simulator A user simulator is an LLM playing the customer. It gets a goal , a persona , and hidden information it reveals only when asked. That last part is critical. If the simulator states everything upfront, you’re back to a single-turn eval. python from dataclasses import dataclass, field @dataclass class UserScenario: goal: str what the user wants, in their terms persona: str tone, patience, expertise opening: str first message hidden facts: dict str, str revealed only if the agent asks constraints: list str = field default factory=list max turns: int = 8 SIMULATOR PROMPT = """You are role-playing a customer contacting billing support. Goal: {goal} Persona: {persona} Facts you know reveal a fact ONLY if the agent asks for it or it is clearly relevant : {facts} Your constraints: {constraints} Rules: - Stay in character. Write short, realistic messages. - Never volunteer facts the agent has not asked about. - If your goal is fully achieved, reply with exactly: DONE - If the agent has clearly failed or is going in circles, reply with exactly: GIVE UP """ class UserSimulator: def init self, scenario: UserScenario, llm : self.s, self.llm = scenario, llm facts = "\n".join f"- {k}: {v}" for k, v in scenario.hidden facts.items self.system = SIMULATOR PROMPT.format goal=scenario.goal, persona=scenario.persona, facts=facts, constraints="; ".join scenario.constraints or "none" def reply self, transcript: list dict - str: Flip roles: the agent's messages are the 'user' from the simulator's view flipped = {"role": "user" if m "role" == "assistant" else "assistant", "content": m "content" } for m in transcript return self.llm system=self.system, messages=flipped Running a conversation: php def run conversation agent, sim: UserSimulator, state - dict: transcript = {"role": "user", "content": sim.s.opening} for turn in range sim.s.max turns : agent msg = agent.respond transcript, state may call tools internally transcript.append {"role": "assistant", "content": agent msg} user msg = sim.reply transcript if user msg.strip in " DONE ", " GIVE UP " : return {"transcript": transcript, "end": user msg.strip , "turns": turn + 1, "final state": state.snapshot } transcript.append {"role": "user", "content": user msg} return {"transcript": transcript, "end": "max turns", "turns": sim.s.max turns, "final state": state.snapshot } An example scenario: UserScenario goal="Get refunded for a duplicate $49 March charge", persona="Frustrated, terse, has already contacted support once", opening="charged twice again. fix it", hidden facts={ "email": "dana@example.com", "month": "March", "previous ticket": "Contacted support 3 days ago, no answer", }, constraints= "Gets annoyed if asked more than two questions", "Does not want to be transferred to a human" , Simulator Pitfalls The simulator is part of the test. If it’s too cooperative, the agent looks better than it is. If it’s inconsistent forgets its own facts, changes its goal , you get noisy, unfair failures. Validate it: - Read 20 simulated conversations by hand before trusting the setup - Check that the simulator never reveals hidden facts unprompted a simple grader can check this - Run each scenario several times. Simulator variance stacks on top of agent variance, so you need more trials than for single-turn cases Lesson 10 . Don’t let the simulator decide success. DONE is a useful stopping signal, but the simulator can be fooled by a confident agent just like a real user. Grade success from state and a separate judge, not from the simulator’s own verdict. Conversation-Level Metrics | Metric | How to measure | |---|---| | Goal success | State assertion refund exists plus judge on the final resolution | | Turns to resolution | Count turns until success; compare to a reference | | Clarification efficiency | Number of agent questions vs number of hidden facts actually needed | | Redundant asks | Agent asked for a fact already provided earlier in the conversation | | Constraint adherence | Judge checks each scenario constraint against the transcript | | Abandonment rate | Fraction ending in GIVE UP or max turns | Redundant asks are checkable without a judge if you tag facts: php def redundant asks transcript: list dict , fact patterns: dict str, str - list str : """fact patterns: fact name - regex that detects the agent asking for it.""" import re provided, redundant = set , for m in transcript: if m "role" == "user": for fact, in fact patterns.items : if fact in message fact, m "content" : provided.add fact else: for fact, pattern in fact patterns.items : if fact in provided and re.search pattern, m "content" , re.I : redundant.append fact return redundant Evaluating Memory Agents with memory across sessions user preferences, prior tickets, account notes need their own tests. Three properties matter: Recall. Information stored in session 1 is used correctly in session 2. Update. When information changes β€œI moved to the annual plan” , the agent uses the new value, not the old one. Forgetting and isolation. Information that should be deleted is gone; one user’s memory never appears in another user’s session. Memory tests are multi-session scenarios: memory case = { "sessions": {"user": "Please always email receipts to billing@acme.example, not my personal address."}, {"user": "Can you resend my April receipt?", "expect tool args": {"send receipt": {"to": "billing@acme.example"}}}, {"user": "Actually, send receipts to finance@acme.example from now on."}, {"user": "Resend May's receipt please.", "expect tool args": {"send receipt": {"to": "finance@acme.example"}}}, } Isolation tests are safety tests. Seed memory for user A, run a session as user B, and assert that none of A’s data appears in B’s tool calls or responses. This belongs in the adversarial suite Lesson 9 as a gate. For long single conversations, also test context-window pressure : pad the conversation with many turns, then check whether an early constraint is still honored. Agents that summarize or truncate history often lose exactly the constraint that mattered. Evaluating Multi-Agent Systems In a supervisor/worker design, a request flows through several agents. Imagine Quill v2: β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” user ─────▢ β”‚ Supervisor β”‚ ─────▢ response β””β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”˜ route β”‚ β”‚ route β”Œβ”€β”€β”€β”€β”€β–Όβ”€β”€β” β”Œβ”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚Billing β”‚ β”‚ Policy/KB β”‚ β”‚ worker β”‚ β”‚ worker β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ A wrong final answer could come from a bad routing decision, a worker error, a lossy handoff, or the supervisor misreading a worker’s result. End-to-end evaluation tells you that it failed. You need attribution to know where . Evaluate Each Role, Then the System 1. Routing accuracy. Step cases for the supervisor: given this request, which worker should handle it? 2. Worker quality. Each worker evaluated in isolation with its own dataset, as a single agent. 3. Handoff fidelity. Does the message the supervisor sends a worker contain everything the worker needs? A judge comparing the original request to the handoff message works well. 4. Synthesis. Given correct worker outputs, does the supervisor compose a correct final response? 5. End to end. The full system on the golden dataset. Attribution With Traces Because each agent’s activity is a subtree of spans Lesson 2 , you can attribute failures automatically: walk the trace and run the per-role graders on each subtree. php def attribute failure trace, case - str: route = first span trace, kind="agent", name="supervisor.route" if route.output "worker" not in case.expected "acceptable workers" : return "routing" worker root = first span trace, kind="agent", name=route.output "worker" if not worker graders pass worker root, case : return f"worker:{route.output 'worker' }" if not handoff complete route.input, worker root.input : return "handoff" return "synthesis" Report failures by attribution bucket: end-to-end pass rate 81% failures by stage: routing 31% worker:billing 22% handoff 38% synthesis 9% A breakdown like that changes what you work on next. Here, improving handoff messages is worth more than improving the billing worker. Multi-Agent Specific Failure Modes - Ping-pong : agents hand a task back and forth without progress. Grade with a max-handoffs budget. - Duplicated work : two workers both perform the same action two refunds . The no-collateral-changes and cardinality gates from earlier lessons catch this. - Authority confusion : a worker takes an action only the supervisor should approve. Check permissions per agent in the trace. Key Takeaways - Many agent failures only appear across turns; single-turn evals can’t see them - A user simulator needs a goal, persona, and hidden facts revealed only when asked - Validate the simulator, and never let it be the judge of success; grade from state and a separate judge - Track turns to resolution, clarification efficiency, redundant asks, constraint adherence, and abandonment - Memory evals test recall , update , and isolation ; isolation is a safety gate - In multi-agent systems, evaluate each role, then attribute failures using trace subtrees Exercise 1. Build a user simulator and write 10 scenarios with hidden facts and constraints for your agent 2. Run each scenario 5 times; report goal success, mean turns, redundant asks, and abandonment 3. Read five transcripts and note any simulator misbehavior; fix the simulator prompt 4. If you have a multi-agent system, implement attribute failure and report failures by stage. If not, write a two-session memory test with an update step.