cd /news/ai-agents/multi-turn-memory-and-multi-agent-ev… Β· home β€Ί topics β€Ί ai-agents β€Ί article
[ARTICLE Β· art-172297] src=superml.org β†— pub= topic=ai-agents verified=true sentiment=Β· neutral

Multi-Turn, Memory, and Multi-Agent Evaluation

A tutorial on multi-turn, memory, and multi-agent evaluation argues that single-turn LLM evals miss conversational failures such as re-asking for information already given, forgetting user constraints, over-asking and under-asking clarifying questions, and state drift between agent and user. The lesson's core technique is an LLM-driven user simulator configured with a goal, persona, hidden facts revealed only when asked, and constraints, plus [DONE] and [GIVE_UP] termination tokens and a max_turns cap of 8 in the sample UserScenario. It also covers conversation-level success and turn-level metrics, memory evaluations for recall, update and forgetting, and attributing failures to specific agents and handoffs in multi-agent systems.

recommendation

Build multi-turn evals with an LLM user simulator that holds hidden facts and reveals them only when asked, and end runs on [DONE], [GIVE_UP] or a max_turns cap to catch over-asking and state drift.

more in playbook: Evals and testing for LLM apps
read8 min views2 publishedNov 18, 2026
Multi-Turn, Memory, and Multi-Agent Evaluation
Image: Superml (auto-discovered)

Β· Agentic AI Β· 7 min read

πŸ“‹ Prerequisites

  • Trajectory and Tool-Use Evaluation (previous lesson)

🎯 What You'll Learn

  • Build an LLM-driven user simulator with goals, personas, and hidden information
  • Define conversation-level success and turn-level metrics for multi-turn agents
  • Design memory evaluations for recall, update, and forgetting
  • Attribute failures to specific agents and handoffs in a multi-agent system

Single-Turn Evals Hide Conversational Failures #

Everything so far assumed one user message per case. Real support conversations aren’t like that. The customer leaves out their email, the agent asks, the customer gives the wrong one, then corrects it. The customer changes their mind halfway through. The customer gets frustrated after the second clarifying question.

Failures that only show up across turns:

  • Asking for information already given , two turns earlier
  • Forgetting a constraint (β€œdon’t escalate me, I’ve been waiting three days”)
  • Over-asking : four clarifying questions when one would do
  • Under-asking : guessing the account when the email was ambiguous
  • State drift : the agent’s understanding of the problem diverges from the user’s

To evaluate these, the test needs a counterpart that talks back.

The User Simulator #

A user simulator is an LLM playing the customer. It gets a goal, a persona, and hidden information it reveals only when asked. That last part is critical. If the simulator states everything upfront, you’re back to a single-turn eval.

from dataclasses import dataclass, field

@dataclass
class UserScenario:
    goal: str                      # what the user wants, in their terms
    persona: str                   # tone, patience, expertise
    opening: str                   # first message
    hidden_facts: dict[str, str]   # revealed only if the agent asks
    constraints: list[str] = field(default_factory=list)
    max_turns: int = 8

SIMULATOR_PROMPT = """You are role-playing a customer contacting billing support.

Goal: {goal}
Persona: {persona}
Facts you know (reveal a fact ONLY if the agent asks for it or it is clearly relevant):
{facts}
Your constraints: {constraints}

Rules:
- Stay in character. Write short, realistic messages.
- Never volunteer facts the agent has not asked about.
- If your goal is fully achieved, reply with exactly: [DONE]
- If the agent has clearly failed or is going in circles, reply with exactly: [GIVE_UP]
"""

class UserSimulator:
    def __init__(self, scenario: UserScenario, llm):
        self.s, self.llm = scenario, llm
        facts = "\n".join(f"- {k}: {v}" for k, v in scenario.hidden_facts.items())
        self.system = SIMULATOR_PROMPT.format(goal=scenario.goal, persona=scenario.persona,
                                              facts=facts,
                                              constraints="; ".join(scenario.constraints) or "none")

    def reply(self, transcript: list[dict]) -> str:
        flipped = [{"role": "user" if m["role"] == "assistant" else "assistant",
                    "content": m["content"]} for m in transcript]
        return self.llm(system=self.system, messages=flipped)

Running a conversation:

def run_conversation(agent, sim: UserSimulator, state) -> dict:
    transcript = [{"role": "user", "content": sim.s.opening}]
    for turn in range(sim.s.max_turns):
        agent_msg = agent.respond(transcript, state)     # may call tools internally
        transcript.append({"role": "assistant", "content": agent_msg})
        user_msg = sim.reply(transcript)
        if user_msg.strip() in ("[DONE]", "[GIVE_UP]"):
            return {"transcript": transcript, "end": user_msg.strip(),
                    "turns": turn + 1, "final_state": state.snapshot()}
        transcript.append({"role": "user", "content": user_msg})
    return {"transcript": transcript, "end": "max_turns",
            "turns": sim.s.max_turns, "final_state": state.snapshot()}

An example scenario:

UserScenario(
    goal="Get refunded for a duplicate $49 March charge",
    persona="Frustrated, terse, has already contacted support once",
    opening="charged twice again. fix it",
    hidden_facts={
        "email": "dana@example.com",
        "month": "March",
        "previous_ticket": "Contacted support 3 days ago, no answer",
    },
    constraints=["Gets annoyed if asked more than two questions",
                 "Does not want to be transferred to a human"],
)

Simulator Pitfalls

The simulator is part of the test. If it’s too cooperative, the agent looks better than it is. If it’s inconsistent (forgets its own facts, changes its goal), you get noisy, unfair failures. Validate it:

  • Read 20 simulated conversations by hand before trusting the setup
  • Check that the simulator never reveals hidden facts unprompted (a simple grader can check this)
  • Run each scenario several times. Simulator variance stacks on top of agent variance, so you need more trials than for single-turn cases (Lesson 10).

Don’t let the simulator decide success. [DONE] is a useful stopping signal, but the simulator can be fooled by a confident agent just like a real user. Grade success from state and a separate judge, not from the simulator’s own verdict.

Conversation-Level Metrics #

Metric How to measure
Goal success State assertion (refund exists) plus judge on the final resolution
Turns to resolution Count turns until success; compare to a reference
Clarification efficiency Number of agent questions vs number of hidden facts actually needed
Redundant asks Agent asked for a fact already provided earlier in the conversation
Constraint adherence Judge checks each scenario constraint against the transcript
Abandonment rate Fraction ending in [GIVE_UP] or max turns

Redundant asks are checkable without a judge if you tag facts:

def redundant_asks(transcript: list[dict], fact_patterns: dict[str, str]) -> list[str]:
    """fact_patterns: fact name -> regex that detects the agent asking for it."""
    import re
    provided, redundant = set(), []
    for m in transcript:
        if m["role"] == "user":
            for fact, _ in fact_patterns.items():
                if fact_in_message(fact, m["content"]):
                    provided.add(fact)
        else:
            for fact, pattern in fact_patterns.items():
                if fact in provided and re.search(pattern, m["content"], re.I):
                    redundant.append(fact)
    return redundant

Evaluating Memory #

Agents with memory across sessions (user preferences, prior tickets, account notes) need their own tests. Three properties matter:

Recall. Information stored in session 1 is used correctly in session 2.

Update. When information changes (β€œI moved to the annual plan”), the agent uses the new value, not the old one.

Forgetting and isolation. Information that should be deleted is gone; one user’s memory never appears in another user’s session.

Memory tests are multi-session scenarios:

memory_case = {
    "sessions": [
        {"user": "Please always email receipts to billing@acme.example, not my personal address."},
        {"user": "Can you resend my April receipt?",
         "expect_tool_args": {"send_receipt": {"to": "billing@acme.example"}}},
        {"user": "Actually, send receipts to finance@acme.example from now on."},
        {"user": "Resend May's receipt please.",
         "expect_tool_args": {"send_receipt": {"to": "finance@acme.example"}}},
    ]
}

Isolation tests are safety tests. Seed memory for user A, run a session as user B, and assert that none of A’s data appears in B’s tool calls or responses. This belongs in the adversarial suite (Lesson 9) as a gate.

For long single conversations, also test context-window pressure: pad the conversation with many turns, then check whether an early constraint is still honored. Agents that summarize or truncate history often lose exactly the constraint that mattered.

Evaluating Multi-Agent Systems #

In a supervisor/worker design, a request flows through several agents. Imagine Quill v2:

             β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
 user ─────▢ β”‚ Supervisor  β”‚ ─────▢ response
             β””β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”˜
        route   β”‚       β”‚   route
          β”Œβ”€β”€β”€β”€β”€β–Όβ”€β”€β” β”Œβ”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
          β”‚Billing β”‚ β”‚ Policy/KB   β”‚
          β”‚ worker β”‚ β”‚   worker    β”‚
          β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

A wrong final answer could come from a bad routing decision, a worker error, a lossy handoff, or the supervisor misreading a worker’s result. End-to-end evaluation tells you that it failed. You need attribution to know where.

Evaluate Each Role, Then the System

  1. Routing accuracy. Step cases for the supervisor: given this request, which worker should handle it?
  2. Worker quality. Each worker evaluated in isolation with its own dataset, as a single agent.
  3. Handoff fidelity. Does the message the supervisor sends a worker contain everything the worker needs? A judge comparing the original request to the handoff message works well.
  4. Synthesis. Given correct worker outputs, does the supervisor compose a correct final response?
  5. End to end. The full system on the golden dataset.

Attribution With Traces

Because each agent’s activity is a subtree of spans (Lesson 2), you can attribute failures automatically: walk the trace and run the per-role graders on each subtree.

def attribute_failure(trace, case) -> str:
    route = first_span(trace, kind="agent", name="supervisor.route")
    if route.output["worker"] not in case.expected["acceptable_workers"]:
        return "routing"
    worker_root = first_span(trace, kind="agent", name=route.output["worker"])
    if not worker_graders_pass(worker_root, case):
        return f"worker:{route.output['worker']}"
    if not handoff_complete(route.input, worker_root.input):
        return "handoff"
    return "synthesis"

Report failures by attribution bucket:

end-to-end pass rate   81%
failures by stage:
  routing              31%
  worker:billing       22%
  handoff              38%
  synthesis             9%

A breakdown like that changes what you work on next. Here, improving handoff messages is worth more than improving the billing worker.

Multi-Agent Specific Failure Modes

  • Ping-pong : agents hand a task back and forth without progress. Grade with a max-handoffs budget.
  • Duplicated work : two workers both perform the same action (two refunds!). The no-collateral-changes and cardinality gates from earlier lessons catch this.
  • Authority confusion : a worker takes an action only the supervisor should approve. Check permissions per agent in the trace.

Key Takeaways #

  • Many agent failures only appear across turns; single-turn evals can’t see them
  • A user simulator needs a goal, persona, andhidden facts revealed only when asked
  • Validate the simulator, and never let it be the judge of success; grade from state and a separate judge
  • Track turns to resolution, clarification efficiency, redundant asks, constraint adherence, and abandonment
  • Memory evals test recall ,update , andisolation ; isolation is a safety gate
  • In multi-agent systems, evaluate each role, then attribute failures using trace subtrees

Exercise #

  1. Build a user simulator and write 10 scenarios with hidden facts and constraints for your agent
  2. Run each scenario 5 times; report goal success, mean turns, redundant asks, and abandonment
  3. Read five transcripts and note any simulator misbehavior; fix the simulator prompt
  4. If you have a multi-agent system, implement attribute_failure and report failures by stage. If not, write a two-session memory test with an update step.
── more in #ai-agents 4 stories Β· sorted by recency
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/multi-turn-memory-an…] indexed:0 read:8min 2026-11-18 Β· β€”