cd /news/ai-agents/trajectory-and-tool-use-evaluation · home › topics › ai-agents › article
[ARTICLE · art-172301] src=superml.org ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Trajectory and Tool-Use Evaluation

A lesson on trajectory and tool-use evaluation shows why outcome-only grading misses agent failures, using three Quill runs that all end in the correct refund: Run B makes seven tool calls with three repeated, and Run C issues the refund without ever confirming the customer or the duplicate. The lesson recommends grading against constraints — required, forbidden, ordering, cardinality, budget and argument checks — rather than exact-match tool sequences, and provides a Python TrajectorySpec dataclass and check_trajectory function that return violations for missing required tools, forbidden calls, ordering breaches, per-tool and total call budgets.

recommendation

Replace exact-match trajectory grading with constraint specs — required, forbidden, ordering, cardinality and max_total call budgets — so valid alternative paths pass while unsafe or wasteful runs like a refund without lookup_customer fail.

more in playbook: Evals and testing for LLM apps
read8 min views2 publishedNov 18, 2026
Trajectory and Tool-Use Evaluation
Image: Superml (auto-discovered)

· Agentic AI · 7 min read

📋 Prerequisites

  • LLM-as-Judge Done Right (previous lesson)

🎯 What You'll Learn

  • Explain why trajectory evaluation catches failures that outcome evaluation misses
  • Grade trajectories with constraints (required, forbidden, ordering, budgets) instead of exact sequences
  • Compute tool-selection and argument-accuracy metrics at the step level
  • Use reference trajectories with flexible matching modes, and judge reasoning where code cannot

Right Answer, Wrong Path #

Consider three Quill runs on the same duplicate-charge case. All three end with the correct refund.

Run A

lookup_customer(email) → get_invoices(C4410) → issue_refund(INV-2291, 49) → respond

Run B

lookup_customer(email) → get_invoices(C4410) → search_kb("duplicate") →
get_invoices(C4410) → search_kb("refund policy") → get_invoices(C4410) →
issue_refund(INV-2291, 49) → respond

Run C

issue_refund(INV-2291, 49) → respond

An outcome grader passes all three. But B is wasteful (seven tool calls, three of them repeated), and C is alarming. It refunded without ever confirming who the customer was or that the invoice was a duplicate. It got lucky because the invoice ID happened to appear in the user’s message. On the next case, that same behavior refunds the wrong person.

Trajectory evaluation is how you see the difference. It answers: was the path acceptable, safe, and efficient?

Why Exact-Match Trajectories Fail #

The naive approach compares the agent’s tool sequence to a single reference sequence. As Lesson 1 showed, there are usually several valid paths. Exact match produces false failures, and a team that sees false failures learns to ignore the eval.

Instead, grade against constraints that capture what actually matters:

Constraint Example for Quill
Required get_invoices must be called before any refund
Forbidden issue_refund must not be called on a read-only question
Ordering lookup_customer must precedeget_invoices
Cardinality issue_refund at most once per case
Budget At most 6 tool calls in total
Argument issue_refund.invoice_id must belong to the looked-up customer

A Constraint-Based Trajectory Grader #

from dataclasses import dataclass, field

@dataclass
class TrajectorySpec:
    required: list[str] = field(default_factory=list)
    forbidden: list[str] = field(default_factory=list)
    order: list[tuple[str, str]] = field(default_factory=list)   # (before, after)
    max_calls: dict[str, int] = field(default_factory=dict)
    max_total: int | None = None

def check_trajectory(calls: list[tuple[str, dict]], spec: TrajectorySpec) -> list[str]:
    names = [n for n, _ in calls]
    violations = []

    for t in spec.required:
        if t not in names:
            violations.append(f"missing required tool: {t}")
    for t in spec.forbidden:
        if t in names:
            violations.append(f"called forbidden tool: {t}")
    for before, after in spec.order:
        if after in names:
            first_after = names.index(after)
            if before not in names[:first_after]:
                violations.append(f"{after} called before {before}")
    for t, n in spec.max_calls.items():
        if names.count(t) > n:
            violations.append(f"{t} called {names.count(t)} times (max {n})")
    if spec.max_total is not None and len(names) > spec.max_total:
        violations.append(f"{len(names)} tool calls (max {spec.max_total})")
    return violations

@dataclass
class TrajectoryGrader:
    default_spec: TrajectorySpec
    name: str = "trajectory"

    def grade(self, case, run) -> Score:
        spec = TrajectorySpec(**case.expected["trajectory"]) \
            if "trajectory" in case.expected else self.default_spec
        violations = check_trajectory(run.trace.tool_calls(), spec)
        n_checks = (len(spec.required) + len(spec.forbidden) + len(spec.order)
                    + len(spec.max_calls) + (spec.max_total is not None))
        value = 1.0 - len(violations) / max(n_checks, 1)
        return Score(self.name, max(value, 0.0), not violations,
                     "; ".join(violations) or "trajectory OK")

A sensible default spec for Quill, applied to every case unless overridden:

QUILL_DEFAULT = TrajectorySpec(
    order=[("lookup_customer", "get_invoices"), ("get_invoices", "issue_refund")],
    max_calls={"issue_refund": 1, "escalate_to_human": 1},
    max_total=8,
)

That default alone would fail Run C (refund before get_invoices) and flag Run B if you tightened max_total to 6.

Redundancy and Efficiency #

Efficiency failures don’t break the outcome, but they cost money and latency, and they’re often an early sign that the agent is confused. Two useful metrics:

def redundancy(calls: list[tuple[str, dict]]) -> float:
    """Fraction of calls that exactly repeat an earlier call (same tool, same args)."""
    seen, repeats = set(), 0
    for name, args in calls:
        key = (name, json.dumps(args, sort_keys=True))
        repeats += key in seen
        seen.add(key)
    return repeats / len(calls) if calls else 0.0

def efficiency(calls, reference_len: int) -> float:
    """1.0 at or below the reference length, decaying as the agent takes more steps."""
    return min(1.0, reference_len / max(len(calls), 1))

Run B’s redundancy is 2/7, about 29%. Track redundancy across versions. A jump usually means the agent stopped trusting its observations, often because of a prompt change or a context-window truncation.

Step-Level Tool Metrics #

The trajectory grader works on whole runs. For diagnosing why trajectories go wrong, measure individual decisions. This is the step layer from Lesson 3.

Build step cases by freezing a conversation at a decision point. You can extract them automatically from reference trajectories: every prefix of a good trajectory is a step case whose expected next action is the next step in that trajectory.

@dataclass
class StepCase:
    id: str
    messages: list[dict]                # history up to the decision
    acceptable_tools: list[str]         # any of these is correct
    expected_args: dict = field(default_factory=dict)   # key -> exact value
    arg_validators: dict = field(default_factory=dict)  # key -> callable

def eval_step(case: StepCase, model_client) -> dict:
    msg = model_client.complete(case.messages, TOOL_SCHEMAS)
    calls = msg.get("tool_calls") or []
    if not calls:
        chosen, args = "(final_answer)", {}
    else:
        chosen, args = calls[0]["name"], calls[0]["arguments"]

    tool_ok = chosen in case.acceptable_tools
    arg_errors = []
    if tool_ok:
        for k, v in case.expected_args.items():
            if args.get(k) != v:
                arg_errors.append(f"{k}={args.get(k)!r}, expected {v!r}")
        for k, fn in case.arg_validators.items():
            if not fn(args.get(k)):
                arg_errors.append(f"{k}={args.get(k)!r} failed validator")
    return {"tool_correct": tool_ok, "args_correct": tool_ok and not arg_errors,
            "chosen": chosen, "arg_errors": arg_errors}

Aggregate into three numbers you should track on every change:

  • Tool selection accuracy : chose an acceptable tool (or correctly chose to answer)
  • Argument accuracy : given the right tool, were the arguments correct
  • Premature-answer rate : answered without calling a tool when one was needed

The last one matters more than it sounds. “Answering from memory instead of looking it up” is one of the most common regressions after a model change. The new model is more confident and skips the lookup.

Argument Matching Modes

Exact match is right for IDs and enums. It’s wrong for free-text arguments like search_kb(query) or escalate_to_human(summary). Use the right mode per argument:

Argument type Matching
IDs, enums, booleans Exact
Amounts, dates Normalized exact or tolerance
Search queries Validator (contains key terms), or check the retrieved result instead
Summaries, free text LLM judge with a narrow rubric, or skip at step level

Reference Trajectories with Flexible Matching #

Sometimes you do want to compare against a reference path, especially for long, procedural tasks. Use flexible matching modes instead of exact equality:

def match_trajectory(actual: list[str], reference: list[str], mode: str) -> bool:
    if mode == "exact":
        return actual == reference
    if mode == "in_order":       # reference is a subsequence of actual
        it = iter(actual)
        return all(step in it for step in reference)
    if mode == "any_order":      # same multiset of required steps
        return all(actual.count(s) >= reference.count(s) for s in set(reference))
    if mode == "superset":       # reference tools all appear, extras allowed
        return set(reference) <= set(actual)
    raise ValueError(mode)

in_order is usually the right default. It requires the essential steps in the right sequence and tolerates extra harmless steps like a KB search. These modes mirror what LangSmith’s and other frameworks’ trajectory evaluators offer, so the concepts transfer directly.

Judging the Reasoning #

Some trajectory properties can’t be checked with constraints:

  • Did the agent recover sensibly after a tool error, or did it retry blindly?
  • When data conflicted (two invoices, different amounts), did it notice and handle the conflict?
  • Was escalating the right call given what the agent knew at that point?

A trajectory judge reads the full sequence of tool calls and observations and applies a narrow rubric. Use the same discipline as Lesson 6: one criterion, binary questions, calibrated against human labels.

Criterion: Error recovery

Q1. Did any tool call return an error?  (If NO, answer N/A for Q2–Q3 and PASS.)
Q2. After the error, did the agent change its approach (different args, a different
    tool, or asking the user), rather than repeating the identical call?
Q3. Did the agent avoid claiming success for the action that errored?

PASS if Q1 is NO, or both Q2 and Q3 are YES.

Putting It in the Scorecard #

quill_scorecard = Scorecard(
    gates=[CompletedGate(), RefundLimitGate(), CorrectCustomerGate()],
    weighted=[
        (RefundOutcomeGrader(), 0.45),
        (TrajectoryGrader(QUILL_DEFAULT), 0.20),
        (RubricJudge("accuracy", ACCURACY_CRITERION, ACCURACY_RUBRIC, judge), 0.25),
        (RubricJudge("clarity", CLARITY_CRITERION, CLARITY_RUBRIC, judge), 0.10),
    ],
)

And report trajectory metrics alongside the pass rate, even when they don’t affect pass/fail:

pass rate            86.0%
tool selection acc.  94.1%   (step suite, 210 decisions)
argument acc.        97.3%
premature answers     2.4%
mean tool calls       4.3    (reference 3.8)
redundancy            6.1%

Key Takeaways #

  • Outcome-only grading misses lucky successes and wasteful paths; trajectory evaluation catches both
  • Grade trajectories against constraints (required, forbidden, ordering, cardinality, budget), not one exact path
  • Track redundancy andefficiency ; jumps signal confusion before outcomes degrade
  • Measure tool selection ,argument accuracy , andpremature-answer rate at the step level
  • Use flexible reference matching (in_order by default); use calibrated judges for recovery and reasoning

Exercise #

  1. Write a default TrajectorySpec for your agent and per-case overrides for 10 cases
  2. Generate at least 50 step cases automatically from reference trajectories
  3. Report tool-selection accuracy, argument accuracy, and premature-answer rate
  4. Swap the agent’s model for a different one (or change the system prompt), and compare all trajectory metrics side by side

Write one paragraph explaining which metric moved the most and what it tells you about the change.

── more in #ai-agents 4 stories · sorted by recency
── more on @quill 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/trajectory-and-tool-…] indexed:0 read:8min 2026-11-18 · —