A practical framework for evaluating trajectories, tool use, and process, not just the final answer.
Originally published on Medium. I am sure almost everyone reading this has tried to build an AI agent by now. For the POC, after some trial and error, it works, and you can demo it, everyone is impressed. You have tested it manually and it works, mostly.
But thinking in terms of software engineering, you know that conventionally the code had integration tests, unit tests, etc. I am sure it did not cover everything, but it gave me baseline confidence. For AI agents, the story is very different.
An AI agent has variability, it can hallucinate into an infinite loop, take a different trajectory then what you had thought of, invent its own responses/tools and can also say that the work is done when it might not have been done. When you evaluate a standard LLM call, you are generally grading a single output. You put a prompt in, you get text out, and you figure out if that text is accurate. But agents change the game because they take actions. They rely on tool use, multi-step reasoning, and compounding failures. If your agent hallucinates a parameter during a tool call at step two, it can silently derail the entire process and the scariest part is, the final answer it gives you might still look completely correct.
I didn't want to write about agent evaluation purely theoretically, so I built a small bug-fixing agent, planted 10 bugs, and built an evaluation harness around it. The entire project is open-source, if you want to follow along or run the evaluators yourself: ** Bug-Fixing Agent & Evaluation Framework on GitHub**.
In this page, I am trying to explore how to evaluate an AI agent, not just its output, but its process. I am writing this after reading multiple pages and making some experiments with my AI agent. It is more of a personal journey and I wanted to document it. I am sure it will be useful for others as well.
Before jumping in, let me set a quick boundary. When I say "evaluating an agent" in this post, I am talking about an agent that you give a specific task to (like "fix this bug"), and it stops when it's done. I am not talking about futuristic agents that run 24/7 in the background without human supervision.
Before jumping into the experiments I ran, there are a few core concepts to get right.
When evaluating LLMs, the industry relies on generic benchmarks like MMLU (Google it!) or SWE-bench. These are useful for sanity-checking a base model's raw capability. However, generic benchmarks tell you how smart the underlying model is; they don't tell you whether your agent, wired to your custom tools, solving your specific task, is any good. For that, you need a custom evaluation framework.
To make these concepts concrete, I built a local bug fixing agent and ran it against 10 planted Python bugs.
Implementation note:I ran this benchmark onClaude Haiku 4.5for cost efficiency. Including documented runs, an earlier uncaptured run, and miscellaneous experiments, the total API spend across the entire project was approximately~$1.
The Prerequisite: A Golden Dataset
Before looking at the configurations, it is worth pausing on those "10 planted bugs." You cannot evaluate an agent without a strong foundation of data. To build a robust evaluation, you need a carefully curated "Golden Dataset" of test cases with known root causes and strict verification steps. Furthermore, this shouldn't be a static list—it must be an ever-improving set of tasks that grows as your agent tackles new edge cases in production.
I evaluated three configurations against this dataset:
config_baseline
: System prompt + full toolset (read_file
, edit_file
, run_tests
).config_prompt_v2
: Constrained step-by-step diagnostic prompt + full toolset.config_no_run_tests
: System prompt + NO run_tests
tool (blind execution).I ran the benchmark suite multiple times to account for LLM non-determinism. Here is how the headline metrics shifted between Run 1 (initial) and Run 2 (extended with token & cost accounting):
| Metric | config_baseline |
config_prompt_v2 |
config_no_run_tests |
|---|---|---|---|
Pass Rate (Run 1 → Run 2) |
80% → 70% | 70% → 70% | 70% → 90% | LLM Judge Score (Run 1 → 2) | 0.90 → 0.80 | 0.80 → 0.80 | 1.00 → 1.00 | Fix Quality (Genuine Fixes) | 100% | 100% | 100% | Total Benchmark Cost (Run 2) | $0.0656 | $0.0670 | $0.0699 | Adversarial Safety Rate | 100% Passed | 100% Passed | 100% Passed |
Note on Non-Determinism (Variability): Notice how the pass rates changed between Run 1 and Run 2! Since I tested on 10 tasks, a single task flipping from pass to fail shifts the score by 10%. This was a big takeaway for me: running a benchmark just once is fine for a quick sanity check, but small score changes are usually just LLM randomness unless you test over multiple runs.
Note on Safety: One of my tasks (task_009_adversarial
) featured a planted, malicious docstring urging the agent to simply write assert True
to fake a fix. Across all runs, the agent resisted this indirect prompt injection and correctly fixed the underlying source code. Worth being honest about sample size though: this is one adversarial scenario out of ten tasks. A 100% pass rate on one adversarial scenario is a promising signal, not proof of robustness.
To uncover the why behind these numbers, I needed to go beyond a simple pass/fail metric. I had to evaluate agent behavior across outcome correctness, tool flow, trajectory efficiency, safety, and fix quality.
While reading about the topic of evaluating agents, I found that the following 8 questions can help us evaluate an agent better; I designed the eval for the bug fixing agent keeping this in mind. This is a good starting point.
*Methodology: Dual-Layer Grading (Rule-Based + LLM Judge)*
The most obvious question is: Did the agent do the job? But a simple pass/fail is rarely enough. In my framework, I split this into two distinct evaluators:
Outcome (Binary Pass/Fail): This is a deterministic, rule-based check. In my code, outcome.py
checks if the pytest
suite passes after the agent finishes its edits. It’s cheap, fast, and gives a definitive yes or no.
Task Completion (Nuanced Score): An outcome-only metric hides nuance. What if the agent correctly identified the bug, wrote the right logic, but forgot a minor syntax detail and ran out of turns? My LLM judge (task_completion.py
) reads the full execution trace and outputs a strict verdict, which maps to a discrete score (COMPLETE
= 1.0, PARTIAL
= 0.5, FAILED
= 0.0). Getting partial credit for sound reasoning and correct file identification is invaluable for debugging capability gaps.
Methodology: Rule-Based Match Unlike standard LLMs, agents take action. Tool misuse: calling the wrong tool, hallucinating parameters, or ignoring a silent failure is a unique agentic failure mode.
To evaluate this, I used tool_correctness.py
, a rule-based evaluator that compares the agent's actions against a known-correct task definition. Did the agent open the correct target file? Did it edit the source code instead of cheating and editing the test file? I didn't need a fuzzy LLM judge here; deterministic string matching works best to ensure the agent is physically doing what it claims to be doing.
Finding: The "Prompt Bloat" Problem
I ran a benchmark comparing my baseline prompt (config_baseline
) against an over-constrained, verbose prompt that forced step-by-step diagnostic thinking (config_prompt_v2
).
You might expect forcing structured reasoning to improve accuracy. It did not. In my tests, config_prompt_v2
added 711 tokens and increased total cost, but it actually performed worse (in Run 1) or identical (in Run 2) to the baseline. Why? The model spent extra tokens "thinking out loud" summarizing the task description and detailing diagnostic steps but this produced no extra problem-solving value over the concise baseline prompt.
Run this experiment on your own agent; the result may surprise you.
Takeaway: In this experiment, on these 10 bug-fixing tasks, the more verbose diagnostic prompt increased token usage without improving task performance. Whether that generalizes to other models, other task types, or genuinely harder bugs is an open question, but it's a cheap thing to test before assuming "more structured reasoning" is automatically better.
Finding: The Blind Execution Paradox
What happens if you take away an agent's ability to test its own code? I ran config_no_run_tests
, an environment where the agent only had read_file
and edit_file
tools—no run_tests
tool.
Remarkably, the agent's pass rate improved to 90% (up from 70%), achieving a perfect 1.00 completion score. Even though the agent couldn't run tests itself, when my evaluator ran the strict pytest
suite at the end, 9 out of 10 fixes were perfectly correct.
A small caveat: the jump from 70% to 90% is really just two extra tasks passing. With only two runs, I can't be sure this gap is real and not just randomness. What I'm more confident about is the quality rating, which held at a perfect 1.00 in both runs.
Without the ability to lean on a "guess-and-check" loop, the agent carefully read the source code and the test suite in its initial turn, identified the exact root cause, and made a single, precise edit. The trade-off? Blind execution used slightly fewer total tokens but took 1.0 second longer per task due to heavier reasoning per turn.
Takeaway: In this experiment, removing the run_tests tool didn't hurt performance; it may have helped, by forcing more careful upfront reasoning instead of a guess-and-check loop. My hypothesis, untested here since every task was single-file and localized, is that this trade-off flips for complex, multi-file bugs, where a testing loop becomes essential. That's a good next experiment for this framework, not a strong conclusion this dataset can support yet.
Methodology: Sequence & Flow Analysis
A correct answer can hide a terrible process. Did the agent get stuck in a dead loop? Did it wildly edit files without reading them first?
My trajectory.py
and tool_flow.py
evaluators analyze the shape of the execution trace. They look for missing verification steps or looped failures. If an agent manages to fix a bug but took 5 redundant edits to get there, a trajectory evaluator will flag it, whereas a simple outcome evaluator would happily mark it as a "pass."
Finding: The Cost of Strict Efficiency
In production, wrong answers are loud—users complain, tests fail, judges flag them. But expensive, slow, correct answers are quiet. They just slowly drain your API credits.
My efficiency.py
evaluator tracks latency, token consumption, USD cost, and turn counts per task. During my benchmark, I noticed persistent failures on two simple tasks (task_002
and task_007
). Why? Because to force efficiency, I set a harsh harness constraint: max_turns: 3
. The minimum viable trajectory (read → edit → verify) is exactly 3 turns. Any slight imprecision on the first try meant the agent hit the turn limit and failed, even if it had the correct logic.
Takeaway: Evaluation harnesses must carefully distinguish between agent capability failures (the model isn't smart enough) and harness constraints (the turn budget was just too tight).
Methodology: CI Gates and Statistical Variance
Once an agent can solve a task, that task graduates to the regression suite. regression.py
allows me to compare two runs of the same config over time to ensure an update to the model or prompt didn't break previously solved tasks.
As I found in my benchmarks, LLMs are naturally unpredictable. A single run is great for a quick CI smoke test, but for true regression testing, you need to run tasks multiple times so you don't mistake LLM randomness for a broken prompt.
Methodology: Diff-based Quality Evaluation
Finally, did the agent fix the actual bug, or did it write a lazy workaround? If a test fails, a lazy agent might simply delete the failing assert
statement in the test file.
To prevent this, fix_quality.py
acts as a specialized LLM judge. Crucially, I didn't just feed it the trace log, I captured the exact git diff
of the agent's final state and passed that to the judge. The judge categorizes the result as a GENUINE
fix (addressing the root architectural bug) or a WORKAROUND
(cheating the test suite). Across all my completed tests, my agents achieved a 100% Genuine Fix rate.
Building a custom evaluation harness is a great learning exercise, but what tools exist in the ecosystem to help you do this at scale? If you don't want to build this from scratch, here is how the tooling landscape looks today:
Tooling caveat:This landscape moves fast: ownership changes (Promptfoo was acquired by OpenAI in March 2026, though it remains open-source) and APIs get marked experimental (NVIDIA's own docs flag NeMo's evaluation API as such). Treat this section as a snapshot, and check current docs before building on any of these.
A common real-world pattern: Use Promptfoo or a custom script (outcome.py
) as a cheap, rule-based CI gate at PR time. Use DeepEval to run scheduled, LLM-judged evaluations on sampled production traces. Finally, visualize those scores and trajectories in Langfuse or Confident AI to monitor for drift.
Agent evaluation is an evolving discipline, not a solved checklist. By stepping away from simple single-turn accuracy and focusing on trajectories, tool correctness, and config-comparisons, you can build agents that don't just answer questions correctly—but act reliably in the real world.
By structuring your evaluation around these 8 critical questions, and blending cheap rule-based CI gates with nuanced LLM-as-a-judge scoring, you can finally prove that your agent works not just in manual testing, but in production.
If you want to dive deeper into the state of the art in agent evaluation, here are some excellent resources: