Your eval suite is green and your agent is still doing something dumb in production. Both of those things can be true at the same time, and the reason is uncomfortable: AI agent evaluation that only scores the final answer is measuring the wrong thing. An agent can pass every check you have while accessing unauthorized resources, leaking private context, or triggering side effects nobody can undo. The final response looks fine, so the run gets marked successful.
Here is the part I think most teams get wrong. We ship agents to production with roughly the same evaluation rigor we would apply to a staging demo, then act surprised when the demo grade harness does not catch production grade failures. This one bit me. Below is what a harness has to look at instead, and what to start logging if you log nothing today.
That is not a contradiction. It is a trajectory problem.
Gartner expects over 40 percent of agentic AI projects to be canceled by the end of 2027, and 32 percent of organizations name quality as the number one deployment barrier. Those numbers are not about models being dumb. They are about teams not being able to tell a good run from a bad one.
Agent failure is usually trajectory level, not output level. A final answer can look completely acceptable while the intermediate steps show wasted cost, unsafe actions, or planning so brittle it only worked by luck. Your scorer never sees any of that, because your scorer only ever sees the last string.
Picture a support agent asked to summarize a customer's order history. It returns a correct summary. Green check. What the trace would have shown you is that it hit an expensive search endpoint eleven times because its first three queries were malformed, then pulled the record from an internal table it was never scoped to read. Correct answer. Terrible run. Your eval suite calls that a pass and moves on, and it will keep calling it a pass every night until a bill or an audit makes it someone's problem.
Call it final answer bias. You grade one output string, so you can only ever detect defects that show up in that string.
Three categories slip straight through:
| Failure | What actually happened | Why the output looks fine |
|---|---|---|
| Unauthorized resource access | Agent queried a datastore outside its scope | The answer it produced was still correct |
| Private context leakage | Sensitive context ended up in a tool argument or downstream call | Leakage happened on the way, not in the reply |
| Irreversible side effects | Agent wrote, deleted, or dispatched something it cannot take back | The confirmation message reads perfectly |
None of those are detectable from a response string, no matter how good your judge prompt is. That is the whole point. A regression suite built only on final accuracy is not neutral, it is actively reassuring you about the exact class of failure it cannot observe. Green means "the last message looked right." It has never meant "nothing bad happened."
There is a published 12 metric evaluation framework for production agents drawn from over 100 deployments. I am not going to recite the twelve names here, because I would be reconstructing them from memory and getting one wrong helps nobody. What I can describe is the shape any serious llm agent evaluation metrics setup has to have.
Start with task outcome, since that is the one you already have. Did the agent do the thing. Keep it, just stop treating it as the whole score.
Then trajectory quality, which asks whether the path was sane. Two runs can land on identical answers and deserve wildly different grades.
Tool call correctness is the one I would add next if I could only add one. Right tool, right arguments, right order, correct handling when the call fails. Most bad trajectories are just a pile of bad tool calls wearing a trench coat.
After that: cost and token efficiency, because agents fail quietly by being expensive long before they fail loudly. Safety and permissions, which is where the three miss categories above finally become measurable. Latency, which nobody cares about until a reasoning loop goes from four steps to nineteen. And human judgment, because some qualities genuinely do not reduce to an automatic scorer, and pretending otherwise just moves the lie somewhere else.
Categories, not a checklist. Fill them in with metrics you can actually compute against your own system.
A trace is the honest version of a run. Every tool call, every argument passed, every intermediate step, every retry, every token spent, in order.
Once you have traces, a tool call audit becomes possible: replay the run and ask whether each call should have happened at all, whether the arguments were well formed, and whether anything in that call touched a resource outside the agent's scope. That is the audit your final answer scorer can never run, because it does not have the material.
This is also why the strongest harnesses stack four things rather than one. Traces tell you what happened. An eval dataset tells you what should have happened on cases you care about. Production monitoring tells you whether live behavior still matches either of those. Human feedback catches what all three miss. Accuracy alone gives you one number and no way to explain it.
If you log nothing today, here is Monday morning. Wrap your tool layer so every invocation writes a record before and after:
type ToolCallRecord = {
runId: string;
step: number;
tool: string;
args: unknown;
ok: boolean;
error?: string;
ms: number;
tokens?: number;
};
export function traced<A extends unknown[], R>(
name: string,
fn: (...args: A) => Promise<R>,
sink: (r: ToolCallRecord) => void,
) {
let step = 0;
return async (runId: string, ...args: A): Promise<R> => {
const started = Date.now();
const current = ++step;
try {
const result = await fn(...args);
sink({ runId, step: current, tool: name, args, ok: true, ms: Date.now() - started });
return result;
} catch (err) {
sink({
runId,
step: current,
tool: name,
args,
ok: false,
error: err instanceof Error ? err.message : String(err),
ms: Date.now() - started,
});
throw err;
}
};
}
That is it. One wrapper, one sink, and suddenly every run has a trajectory you can grade instead of a single string you can only trust.
Teams spend weeks picking a harness and an afternoon writing the cases. It should be the other way around. Agent datasets almost never capture real production failure modes, which is exactly why a demo grade suite passes everything.
The fix is unglamorous: harvest. Every production run that went sideways, whether a user complained, a retry storm showed up in the logs, or a trace looked wrong on review, becomes a case. Freeze the inputs, record what the trajectory should have looked like, drop it into the regression suite. Do that for a month and you have an eval dataset your competitors cannot copy, because it is made of your own scar tissue.
If you want the fuller version of how these pieces fit together, I wrote up an AI agent evaluation framework with the layering in more detail, and a companion piece on LLM agent evaluation in production.
How do you evaluate AI agents in production?
Score the trajectory, not just the reply. Combine traces of every tool call with an eval dataset built from real production failures, live monitoring of the running system, and periodic human review. Task outcome stays in the mix, it just stops being the only signal. The goal is being able to explain why a run passed, not only that it did.
What metrics should an agent eval harness measure?
Cover seven areas rather than one number: task outcome, trajectory quality, tool call correctness, cost and token efficiency, safety and permissions, latency, and human judgment. If you can only add one thing to an existing accuracy check, add tool call correctness. Most bad trajectories are a sequence of bad tool calls, and that metric surfaces them immediately.
Why do agents pass evals but fail in production?
Because evals grade the last message and production grades everything else. An agent can reach a correct answer through an expensive, unsafe, or barely working path, and a final answer scorer has no way to see any of it. Add the fact that eval datasets rarely contain real production failures, and a green suite tells you very little.
If you want a deeper look at layering traces, datasets, and monitoring together, I cover it in more detail on my site.
If you want this wired up on your own site end to end, that is exactly the kind of work I take on.
Drop a comment if your setup looks different. Curious what variations people are running in production.