Trajectory and Tool-Use Evaluation A lesson on trajectory and tool-use evaluation shows why outcome-only grading misses agent failures, using three Quill runs that all end in the correct refund: Run B makes seven tool calls with three repeated, and Run C issues the refund without ever confirming the customer or the duplicate. The lesson recommends grading against constraints — required, forbidden, ordering, cardinality, budget and argument checks — rather than exact-match tool sequences, and provides a Python TrajectorySpec dataclass and check_trajectory function that return violations for missing required tools, forbidden calls, ordering breaches, per-tool and total call budgets. · Agentic AI · 7 min read 📋 Prerequisites - LLM-as-Judge Done Right previous lesson 🎯 What You'll Learn - Explain why trajectory evaluation catches failures that outcome evaluation misses - Grade trajectories with constraints required, forbidden, ordering, budgets instead of exact sequences - Compute tool-selection and argument-accuracy metrics at the step level - Use reference trajectories with flexible matching modes, and judge reasoning where code cannot Right Answer, Wrong Path Consider three Quill runs on the same duplicate-charge case. All three end with the correct refund. Run A lookup customer email → get invoices C4410 → issue refund INV-2291, 49 → respond Run B lookup customer email → get invoices C4410 → search kb "duplicate" → get invoices C4410 → search kb "refund policy" → get invoices C4410 → issue refund INV-2291, 49 → respond Run C issue refund INV-2291, 49 → respond An outcome grader passes all three. But B is wasteful seven tool calls, three of them repeated , and C is alarming. It refunded without ever confirming who the customer was or that the invoice was a duplicate. It got lucky because the invoice ID happened to appear in the user’s message. On the next case, that same behavior refunds the wrong person. Trajectory evaluation is how you see the difference. It answers: was the path acceptable, safe, and efficient? Why Exact-Match Trajectories Fail The naive approach compares the agent’s tool sequence to a single reference sequence. As Lesson 1 showed, there are usually several valid paths. Exact match produces false failures, and a team that sees false failures learns to ignore the eval. Instead, grade against constraints that capture what actually matters: | Constraint | Example for Quill | |---|---| | Required | get invoices must be called before any refund | | Forbidden | issue refund must not be called on a read-only question | | Ordering | lookup customer must precede get invoices | | Cardinality | issue refund at most once per case | | Budget | At most 6 tool calls in total | | Argument | issue refund.invoice id must belong to the looked-up customer | A Constraint-Based Trajectory Grader python from dataclasses import dataclass, field @dataclass class TrajectorySpec: required: list str = field default factory=list forbidden: list str = field default factory=list order: list tuple str, str = field default factory=list before, after max calls: dict str, int = field default factory=dict max total: int | None = None def check trajectory calls: list tuple str, dict , spec: TrajectorySpec - list str : names = n for n, in calls violations = for t in spec.required: if t not in names: violations.append f"missing required tool: {t}" for t in spec.forbidden: if t in names: violations.append f"called forbidden tool: {t}" for before, after in spec.order: if after in names: first after = names.index after if before not in names :first after : violations.append f"{after} called before {before}" for t, n in spec.max calls.items : if names.count t n: violations.append f"{t} called {names.count t } times max {n} " if spec.max total is not None and len names spec.max total: violations.append f"{len names } tool calls max {spec.max total} " return violations @dataclass class TrajectoryGrader: default spec: TrajectorySpec name: str = "trajectory" def grade self, case, run - Score: spec = TrajectorySpec case.expected "trajectory" \ if "trajectory" in case.expected else self.default spec violations = check trajectory run.trace.tool calls , spec n checks = len spec.required + len spec.forbidden + len spec.order + len spec.max calls + spec.max total is not None value = 1.0 - len violations / max n checks, 1 return Score self.name, max value, 0.0 , not violations, "; ".join violations or "trajectory OK" A sensible default spec for Quill, applied to every case unless overridden: QUILL DEFAULT = TrajectorySpec order= "lookup customer", "get invoices" , "get invoices", "issue refund" , max calls={"issue refund": 1, "escalate to human": 1}, max total=8, That default alone would fail Run C refund before get invoices and flag Run B if you tightened max total to 6. Redundancy and Efficiency Efficiency failures don’t break the outcome, but they cost money and latency, and they’re often an early sign that the agent is confused. Two useful metrics: php def redundancy calls: list tuple str, dict - float: """Fraction of calls that exactly repeat an earlier call same tool, same args .""" seen, repeats = set , 0 for name, args in calls: key = name, json.dumps args, sort keys=True repeats += key in seen seen.add key return repeats / len calls if calls else 0.0 def efficiency calls, reference len: int - float: """1.0 at or below the reference length, decaying as the agent takes more steps.""" return min 1.0, reference len / max len calls , 1 Run B’s redundancy is 2/7, about 29%. Track redundancy across versions. A jump usually means the agent stopped trusting its observations, often because of a prompt change or a context-window truncation. Step-Level Tool Metrics The trajectory grader works on whole runs. For diagnosing why trajectories go wrong, measure individual decisions. This is the step layer from Lesson 3. Build step cases by freezing a conversation at a decision point. You can extract them automatically from reference trajectories: every prefix of a good trajectory is a step case whose expected next action is the next step in that trajectory. @dataclass class StepCase: id: str messages: list dict history up to the decision acceptable tools: list str any of these is correct expected args: dict = field default factory=dict key - exact value arg validators: dict = field default factory=dict key - callable def eval step case: StepCase, model client - dict: msg = model client.complete case.messages, TOOL SCHEMAS calls = msg.get "tool calls" or if not calls: chosen, args = " final answer ", {} else: chosen, args = calls 0 "name" , calls 0 "arguments" tool ok = chosen in case.acceptable tools arg errors = if tool ok: for k, v in case.expected args.items : if args.get k = v: arg errors.append f"{k}={args.get k r}, expected {v r}" for k, fn in case.arg validators.items : if not fn args.get k : arg errors.append f"{k}={args.get k r} failed validator" return {"tool correct": tool ok, "args correct": tool ok and not arg errors, "chosen": chosen, "arg errors": arg errors} Aggregate into three numbers you should track on every change: - Tool selection accuracy : chose an acceptable tool or correctly chose to answer - Argument accuracy : given the right tool, were the arguments correct - Premature-answer rate : answered without calling a tool when one was needed The last one matters more than it sounds. “Answering from memory instead of looking it up” is one of the most common regressions after a model change. The new model is more confident and skips the lookup. Argument Matching Modes Exact match is right for IDs and enums. It’s wrong for free-text arguments like search kb query or escalate to human summary . Use the right mode per argument: | Argument type | Matching | |---|---| | IDs, enums, booleans | Exact | | Amounts, dates | Normalized exact or tolerance | | Search queries | Validator contains key terms , or check the retrieved result instead | | Summaries, free text | LLM judge with a narrow rubric, or skip at step level | Reference Trajectories with Flexible Matching Sometimes you do want to compare against a reference path, especially for long, procedural tasks. Use flexible matching modes instead of exact equality: php def match trajectory actual: list str , reference: list str , mode: str - bool: if mode == "exact": return actual == reference if mode == "in order": reference is a subsequence of actual it = iter actual return all step in it for step in reference if mode == "any order": same multiset of required steps return all actual.count s = reference.count s for s in set reference if mode == "superset": reference tools all appear, extras allowed return set reference <= set actual raise ValueError mode in order is usually the right default. It requires the essential steps in the right sequence and tolerates extra harmless steps like a KB search. These modes mirror what LangSmith’s and other frameworks’ trajectory evaluators offer, so the concepts transfer directly. Judging the Reasoning Some trajectory properties can’t be checked with constraints: - Did the agent recover sensibly after a tool error, or did it retry blindly? - When data conflicted two invoices, different amounts , did it notice and handle the conflict? - Was escalating the right call given what the agent knew at that point? A trajectory judge reads the full sequence of tool calls and observations and applies a narrow rubric. Use the same discipline as Lesson 6: one criterion, binary questions, calibrated against human labels. Criterion: Error recovery Q1. Did any tool call return an error? If NO, answer N/A for Q2–Q3 and PASS. Q2. After the error, did the agent change its approach different args, a different tool, or asking the user , rather than repeating the identical call? Q3. Did the agent avoid claiming success for the action that errored? PASS if Q1 is NO, or both Q2 and Q3 are YES. Putting It in the Scorecard quill scorecard = Scorecard gates= CompletedGate , RefundLimitGate , CorrectCustomerGate , weighted= RefundOutcomeGrader , 0.45 , TrajectoryGrader QUILL DEFAULT , 0.20 , RubricJudge "accuracy", ACCURACY CRITERION, ACCURACY RUBRIC, judge , 0.25 , RubricJudge "clarity", CLARITY CRITERION, CLARITY RUBRIC, judge , 0.10 , , And report trajectory metrics alongside the pass rate, even when they don’t affect pass/fail: pass rate 86.0% tool selection acc. 94.1% step suite, 210 decisions argument acc. 97.3% premature answers 2.4% mean tool calls 4.3 reference 3.8 redundancy 6.1% Key Takeaways - Outcome-only grading misses lucky successes and wasteful paths; trajectory evaluation catches both - Grade trajectories against constraints required, forbidden, ordering, cardinality, budget , not one exact path - Track redundancy and efficiency ; jumps signal confusion before outcomes degrade - Measure tool selection , argument accuracy , and premature-answer rate at the step level - Use flexible reference matching in order by default ; use calibrated judges for recovery and reasoning Exercise 1. Write a default TrajectorySpec for your agent and per-case overrides for 10 cases 2. Generate at least 50 step cases automatically from reference trajectories 3. Report tool-selection accuracy, argument accuracy, and premature-answer rate 4. Swap the agent’s model for a different one or change the system prompt , and compare all trajectory metrics side by side Write one paragraph explaining which metric moved the most and what it tells you about the change.