{"slug": "trajectory-and-tool-use-evaluation", "title": "Trajectory and Tool-Use Evaluation", "summary": "A lesson on trajectory and tool-use evaluation shows why outcome-only grading misses agent failures, using three Quill runs that all end in the correct refund: Run B makes seven tool calls with three repeated, and Run C issues the refund without ever confirming the customer or the duplicate. The lesson recommends grading against constraints — required, forbidden, ordering, cardinality, budget and argument checks — rather than exact-match tool sequences, and provides a Python TrajectorySpec dataclass and check_trajectory function that return violations for missing required tools, forbidden calls, ordering breaches, per-tool and total call budgets.", "body_md": "· Agentic AI · 7 min read\n\n### 📋 Prerequisites\n\n- LLM-as-Judge Done Right (previous lesson)\n\n### 🎯 What You'll Learn\n\n- Explain why trajectory evaluation catches failures that outcome evaluation misses\n- Grade trajectories with constraints (required, forbidden, ordering, budgets) instead of exact sequences\n- Compute tool-selection and argument-accuracy metrics at the step level\n- Use reference trajectories with flexible matching modes, and judge reasoning where code cannot\n\n## Right Answer, Wrong Path\n\nConsider three Quill runs on the same duplicate-charge case. All three end with the correct refund.\n\n**Run A**\n\n```\nlookup_customer(email) → get_invoices(C4410) → issue_refund(INV-2291, 49) → respond\n```\n\n**Run B**\n\n```\nlookup_customer(email) → get_invoices(C4410) → search_kb(\"duplicate\") →\nget_invoices(C4410) → search_kb(\"refund policy\") → get_invoices(C4410) →\nissue_refund(INV-2291, 49) → respond\n```\n\n**Run C**\n\n```\nissue_refund(INV-2291, 49) → respond\n```\n\nAn outcome grader passes all three. But B is wasteful (seven tool calls, three of them repeated), and C is alarming. It refunded without ever confirming who the customer was or that the invoice was a duplicate. It got lucky because the invoice ID happened to appear in the user’s message. On the next case, that same behavior refunds the wrong person.\n\n**Trajectory evaluation** is how you see the difference. It answers: *was the path acceptable, safe, and efficient?*\n\n## Why Exact-Match Trajectories Fail\n\nThe naive approach compares the agent’s tool sequence to a single reference sequence. As Lesson 1 showed, there are usually several valid paths. Exact match produces false failures, and a team that sees false failures learns to ignore the eval.\n\nInstead, grade against **constraints** that capture what actually matters:\n\n| Constraint | Example for Quill | \n|---|---|\n| **Required** | `get_invoices` must be called before any refund | \n| **Forbidden** | `issue_refund` must not be called on a read-only question | \n| **Ordering** | `lookup_customer` must precede`get_invoices` | \n| **Cardinality** | `issue_refund` at most once per case | \n| **Budget** | At most 6 tool calls in total | \n| **Argument** | `issue_refund.invoice_id` must belong to the looked-up customer | \n\n## A Constraint-Based Trajectory Grader\n\n``` python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass TrajectorySpec:\n    required: list[str] = field(default_factory=list)\n    forbidden: list[str] = field(default_factory=list)\n    order: list[tuple[str, str]] = field(default_factory=list)   # (before, after)\n    max_calls: dict[str, int] = field(default_factory=dict)\n    max_total: int | None = None\n\ndef check_trajectory(calls: list[tuple[str, dict]], spec: TrajectorySpec) -> list[str]:\n    names = [n for n, _ in calls]\n    violations = []\n\n    for t in spec.required:\n        if t not in names:\n            violations.append(f\"missing required tool: {t}\")\n    for t in spec.forbidden:\n        if t in names:\n            violations.append(f\"called forbidden tool: {t}\")\n    for before, after in spec.order:\n        if after in names:\n            first_after = names.index(after)\n            if before not in names[:first_after]:\n                violations.append(f\"{after} called before {before}\")\n    for t, n in spec.max_calls.items():\n        if names.count(t) > n:\n            violations.append(f\"{t} called {names.count(t)} times (max {n})\")\n    if spec.max_total is not None and len(names) > spec.max_total:\n        violations.append(f\"{len(names)} tool calls (max {spec.max_total})\")\n    return violations\n\n@dataclass\nclass TrajectoryGrader:\n    default_spec: TrajectorySpec\n    name: str = \"trajectory\"\n\n    def grade(self, case, run) -> Score:\n        spec = TrajectorySpec(**case.expected[\"trajectory\"]) \\\n            if \"trajectory\" in case.expected else self.default_spec\n        violations = check_trajectory(run.trace.tool_calls(), spec)\n        n_checks = (len(spec.required) + len(spec.forbidden) + len(spec.order)\n                    + len(spec.max_calls) + (spec.max_total is not None))\n        value = 1.0 - len(violations) / max(n_checks, 1)\n        return Score(self.name, max(value, 0.0), not violations,\n                     \"; \".join(violations) or \"trajectory OK\")\n```\n\nA sensible default spec for Quill, applied to every case unless overridden:\n\n```\nQUILL_DEFAULT = TrajectorySpec(\n    order=[(\"lookup_customer\", \"get_invoices\"), (\"get_invoices\", \"issue_refund\")],\n    max_calls={\"issue_refund\": 1, \"escalate_to_human\": 1},\n    max_total=8,\n)\n```\n\nThat default alone would fail Run C (refund before `get_invoices`) and flag Run B if you tightened `max_total` to 6.\n\n## Redundancy and Efficiency\n\nEfficiency failures don’t break the outcome, but they cost money and latency, and they’re often an early sign that the agent is confused. Two useful metrics:\n\n``` php\ndef redundancy(calls: list[tuple[str, dict]]) -> float:\n    \"\"\"Fraction of calls that exactly repeat an earlier call (same tool, same args).\"\"\"\n    seen, repeats = set(), 0\n    for name, args in calls:\n        key = (name, json.dumps(args, sort_keys=True))\n        repeats += key in seen\n        seen.add(key)\n    return repeats / len(calls) if calls else 0.0\n\ndef efficiency(calls, reference_len: int) -> float:\n    \"\"\"1.0 at or below the reference length, decaying as the agent takes more steps.\"\"\"\n    return min(1.0, reference_len / max(len(calls), 1))\n```\n\nRun B’s redundancy is 2/7, about 29%. Track redundancy across versions. A jump usually means the agent stopped trusting its observations, often because of a prompt change or a context-window truncation.\n\n## Step-Level Tool Metrics\n\nThe trajectory grader works on whole runs. For diagnosing *why* trajectories go wrong, measure individual decisions. This is the **step layer** from Lesson 3.\n\nBuild step cases by freezing a conversation at a decision point. You can extract them automatically from reference trajectories: every prefix of a good trajectory is a step case whose expected next action is the next step in that trajectory.\n\n```\n@dataclass\nclass StepCase:\n    id: str\n    messages: list[dict]                # history up to the decision\n    acceptable_tools: list[str]         # any of these is correct\n    expected_args: dict = field(default_factory=dict)   # key -> exact value\n    arg_validators: dict = field(default_factory=dict)  # key -> callable\n\ndef eval_step(case: StepCase, model_client) -> dict:\n    msg = model_client.complete(case.messages, TOOL_SCHEMAS)\n    calls = msg.get(\"tool_calls\") or []\n    if not calls:\n        chosen, args = \"(final_answer)\", {}\n    else:\n        chosen, args = calls[0][\"name\"], calls[0][\"arguments\"]\n\n    tool_ok = chosen in case.acceptable_tools\n    arg_errors = []\n    if tool_ok:\n        for k, v in case.expected_args.items():\n            if args.get(k) != v:\n                arg_errors.append(f\"{k}={args.get(k)!r}, expected {v!r}\")\n        for k, fn in case.arg_validators.items():\n            if not fn(args.get(k)):\n                arg_errors.append(f\"{k}={args.get(k)!r} failed validator\")\n    return {\"tool_correct\": tool_ok, \"args_correct\": tool_ok and not arg_errors,\n            \"chosen\": chosen, \"arg_errors\": arg_errors}\n```\n\nAggregate into three numbers you should track on every change:\n\n- **Tool selection accuracy** : chose an acceptable tool (or correctly chose to answer)\n- **Argument accuracy** : given the right tool, were the arguments correct\n- **Premature-answer rate** : answered without calling a tool when one was needed\n\nThe last one matters more than it sounds. “Answering from memory instead of looking it up” is one of the most common regressions after a model change. The new model is more confident and skips the lookup.\n\n### Argument Matching Modes\n\nExact match is right for IDs and enums. It’s wrong for free-text arguments like `search_kb(query)` or `escalate_to_human(summary)`. Use the right mode per argument:\n\n| Argument type | Matching | \n|---|---|\n| IDs, enums, booleans | Exact | \n| Amounts, dates | Normalized exact or tolerance | \n| Search queries | Validator (contains key terms), or check the retrieved result instead | \n| Summaries, free text | LLM judge with a narrow rubric, or skip at step level | \n\n## Reference Trajectories with Flexible Matching\n\nSometimes you do want to compare against a reference path, especially for long, procedural tasks. Use flexible matching modes instead of exact equality:\n\n``` php\ndef match_trajectory(actual: list[str], reference: list[str], mode: str) -> bool:\n    if mode == \"exact\":\n        return actual == reference\n    if mode == \"in_order\":       # reference is a subsequence of actual\n        it = iter(actual)\n        return all(step in it for step in reference)\n    if mode == \"any_order\":      # same multiset of required steps\n        return all(actual.count(s) >= reference.count(s) for s in set(reference))\n    if mode == \"superset\":       # reference tools all appear, extras allowed\n        return set(reference) <= set(actual)\n    raise ValueError(mode)\n```\n\n`in_order` is usually the right default. It requires the essential steps in the right sequence and tolerates extra harmless steps like a KB search. These modes mirror what LangSmith’s and other frameworks’ trajectory evaluators offer, so the concepts transfer directly.\n\n## Judging the Reasoning\n\nSome trajectory properties can’t be checked with constraints:\n\n- Did the agent **recover sensibly** after a tool error, or did it retry blindly?\n- When data conflicted (two invoices, different amounts), did it **notice and handle** the conflict?\n- Was escalating the right call given what the agent knew at that point?\n\nA trajectory judge reads the full sequence of tool calls and observations and applies a narrow rubric. Use the same discipline as Lesson 6: one criterion, binary questions, calibrated against human labels.\n\n```\nCriterion: Error recovery\n\nQ1. Did any tool call return an error?  (If NO, answer N/A for Q2–Q3 and PASS.)\nQ2. After the error, did the agent change its approach (different args, a different\n    tool, or asking the user), rather than repeating the identical call?\nQ3. Did the agent avoid claiming success for the action that errored?\n\nPASS if Q1 is NO, or both Q2 and Q3 are YES.\n```\n\n## Putting It in the Scorecard\n\n```\nquill_scorecard = Scorecard(\n    gates=[CompletedGate(), RefundLimitGate(), CorrectCustomerGate()],\n    weighted=[\n        (RefundOutcomeGrader(), 0.45),\n        (TrajectoryGrader(QUILL_DEFAULT), 0.20),\n        (RubricJudge(\"accuracy\", ACCURACY_CRITERION, ACCURACY_RUBRIC, judge), 0.25),\n        (RubricJudge(\"clarity\", CLARITY_CRITERION, CLARITY_RUBRIC, judge), 0.10),\n    ],\n)\n```\n\nAnd report trajectory metrics alongside the pass rate, even when they don’t affect pass/fail:\n\n```\npass rate            86.0%\ntool selection acc.  94.1%   (step suite, 210 decisions)\nargument acc.        97.3%\npremature answers     2.4%\nmean tool calls       4.3    (reference 3.8)\nredundancy            6.1%\n```\n\n## Key Takeaways\n\n- Outcome-only grading misses lucky successes and wasteful paths; trajectory evaluation catches both\n- Grade trajectories against **constraints** (required, forbidden, ordering, cardinality, budget), not one exact path\n- Track **redundancy** and**efficiency** ; jumps signal confusion before outcomes degrade\n- Measure **tool selection** ,**argument accuracy** , and**premature-answer rate** at the step level\n- Use flexible reference matching (`in_order` by default); use calibrated judges for recovery and reasoning\n\n## Exercise\n\n1. Write a default `TrajectorySpec` for your agent and per-case overrides for 10 cases\n2. Generate at least 50 step cases automatically from reference trajectories\n3. Report tool-selection accuracy, argument accuracy, and premature-answer rate\n4. Swap the agent’s model for a different one (or change the system prompt), and compare all trajectory metrics side by side\n\nWrite one paragraph explaining which metric moved the most and what it tells you about the change.", "url": "https://wpnews.pro/news/trajectory-and-tool-use-evaluation", "canonical_source": "https://superml.org/tutorials/agent-eval-trajectory-tool-use", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-19 03:43:44.457235+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-research"], "entities": ["Quill", "TrajectorySpec", "TrajectoryGrader", "check_trajectory"], "also_reported_by": [], "takeaway": "Grade agent traces with constraint checks (required, forbidden, ordering, per-tool and total call budgets) instead of exact-match sequences, since a grader that passes a run issuing a refund without lookup_customer or get_invoices hides unsafe behavior.", "recommendation": {"advice": "Replace exact-match trajectory grading with constraint specs — required, forbidden, ordering, cardinality and max_total call budgets — so valid alternative paths pass while unsafe or wasteful runs like a refund without lookup_customer fail.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/trajectory-and-tool-use-evaluation", "markdown": "https://wpnews.pro/news/trajectory-and-tool-use-evaluation.md", "text": "https://wpnews.pro/news/trajectory-and-tool-use-evaluation.txt", "jsonld": "https://wpnews.pro/news/trajectory-and-tool-use-evaluation.jsonld"}}