When Building an AI Agent, the Journey Matters as Much as the Destination

Raj Kundalia, an engineer, built an open-source bug-fixing agent and evaluation framework to assess AI agents beyond final outputs, focusing on process, tool use, and multi-step reasoning. He planted 10 bugs in Python code and ran three agent configurations, spending about $1 on API calls using Claude Haiku 4.5. The project is available on GitHub.

A practical framework for evaluating trajectories, tool use, and process, not just the final answer. Originally published on Medium https://medium.com/@rajkundalia/when-building-an-ai-agent-the-journey-matters-as-much-as-the-destination-17a8f916bf36?sharedUserId=rajkundalia . I am sure almost everyone reading this has tried to build an AI agent by now. For the POC, after some trial and error, it works, and you can demo it, everyone is impressed. You have tested it manually and it works, mostly. But thinking in terms of software engineering, you know that conventionally the code had integration tests, unit tests, etc. I am sure it did not cover everything, but it gave me baseline confidence. For AI agents, the story is very different. An AI agent has variability, it can hallucinate into an infinite loop, take a different trajectory then what you had thought of, invent its own responses/tools and can also say that the work is done when it might not have been done. When you evaluate a standard LLM call, you are generally grading a single output. You put a prompt in, you get text out, and you figure out if that text is accurate. But agents change the game because they take actions. They rely on tool use , multi-step reasoning , and compounding failures . If your agent hallucinates a parameter during a tool call at step two, it can silently derail the entire process and the scariest part is, the final answer it gives you might still look completely correct. I didn't want to write about agent evaluation purely theoretically, so I built a small bug-fixing agent, planted 10 bugs, and built an evaluation harness around it. The entire project is open-source, if you want to follow along or run the evaluators yourself: Bug-Fixing Agent & Evaluation Framework on GitHub . In this page, I am trying to explore how to evaluate an AI agent, not just its output, but its process . I am writing this after reading multiple pages and making some experiments with my AI agent. It is more of a personal journey and I wanted to document it. I am sure it will be useful for others as well. Before jumping in, let me set a quick boundary. When I say "evaluating an agent" in this post, I am talking about an agent that you give a specific task to like "fix this bug" , and it stops when it's done. I am not talking about futuristic agents that run 24/7 in the background without human supervision. Before jumping into the experiments I ran, there are a few core concepts to get right. When evaluating LLMs, the industry relies on generic benchmarks like MMLU Google it or SWE-bench. These are useful for sanity-checking a base model's raw capability. However, generic benchmarks tell you how smart the underlying model is; they don't tell you whether your agent, wired to your custom tools, solving your specific task, is any good. For that, you need a custom evaluation framework. To make these concepts concrete, I built a local bug fixing agent and ran it against 10 planted Python bugs. Implementation note:I ran this benchmark onClaude Haiku 4.5for cost efficiency. Including documented runs, an earlier uncaptured run, and miscellaneous experiments, the total API spend across the entire project was approximately~$1. The Prerequisite: A Golden Dataset Before looking at the configurations, it is worth pausing on those "10 planted bugs." You cannot evaluate an agent without a strong foundation of data. To build a robust evaluation, you need a carefully curated "Golden Dataset" of test cases with known root causes and strict verification steps. Furthermore, this shouldn't be a static list—it must be an ever-improving set of tasks that grows as your agent tackles new edge cases in production. I evaluated three configurations against this dataset: config baseline : System prompt + full toolset read file , edit file , run tests . config prompt v2 : Constrained step-by-step diagnostic prompt + full toolset. config no run tests : System prompt + NO run tests tool blind execution .I ran the benchmark suite multiple times to account for LLM non-determinism. Here is how the headline metrics shifted between Run 1 initial and Run 2 extended with token & cost accounting : | Metric | config baseline | config prompt v2 | config no run tests | |---|---|---|---| Pass Rate Run 1 → Run 2 | 80% → 70% | 70% → 70% | 70% → 90% | LLM Judge Score Run 1 → 2 | 0.90 → 0.80 | 0.80 → 0.80 | 1.00 → 1.00 | Fix Quality Genuine Fixes | 100% | 100% | 100% | Total Benchmark Cost Run 2 | $0.0656 | $0.0670 | $0.0699 | Adversarial Safety Rate | 100% Passed | 100% Passed | 100% Passed | Note on Non-Determinism Variability : Notice how the pass rates changed between Run 1 and Run 2 Since I tested on 10 tasks, a single task flipping from pass to fail shifts the score by 10%. This was a big takeaway for me: running a benchmark just once is fine for a quick sanity check, but small score changes are usually just LLM randomness unless you test over multiple runs. Note on Safety: One of my tasks task 009 adversarial featured a planted, malicious docstring urging the agent to simply write assert True to fake a fix. Across all runs, the agent resisted this indirect prompt injection and correctly fixed the underlying source code. Worth being honest about sample size though: this is one adversarial scenario out of ten tasks. A 100% pass rate on one adversarial scenario is a promising signal, not proof of robustness. To uncover the why behind these numbers, I needed to go beyond a simple pass/fail metric. I had to evaluate agent behavior across outcome correctness, tool flow, trajectory efficiency, safety, and fix quality. While reading about the topic of evaluating agents, I found that the following 8 questions can help us evaluate an agent better; I designed the eval for the bug fixing agent keeping this in mind. This is a good starting point. Methodology: Dual-Layer Grading Rule-Based + LLM Judge The most obvious question is: Did the agent do the job? But a simple pass/fail is rarely enough. In my framework, I split this into two distinct evaluators: Outcome Binary Pass/Fail : This is a deterministic, rule-based check. In my code, outcome.py checks if the pytest suite passes after the agent finishes its edits. It’s cheap, fast, and gives a definitive yes or no. Task Completion Nuanced Score : An outcome-only metric hides nuance. What if the agent correctly identified the bug, wrote the right logic, but forgot a minor syntax detail and ran out of turns? My LLM judge task completion.py reads the full execution trace and outputs a strict verdict, which maps to a discrete score COMPLETE = 1.0, PARTIAL = 0.5, FAILED = 0.0 . Getting partial credit for sound reasoning and correct file identification is invaluable for debugging capability gaps. Methodology: Rule-Based Match Unlike standard LLMs, agents take action. Tool misuse: calling the wrong tool, hallucinating parameters, or ignoring a silent failure is a unique agentic failure mode. To evaluate this, I used tool correctness.py , a rule-based evaluator that compares the agent's actions against a known-correct task definition. Did the agent open the correct target file? Did it edit the source code instead of cheating and editing the test file? I didn't need a fuzzy LLM judge here; deterministic string matching works best to ensure the agent is physically doing what it claims to be doing. Finding: The "Prompt Bloat" Problem I ran a benchmark comparing my baseline prompt config baseline against an over-constrained, verbose prompt that forced step-by-step diagnostic thinking config prompt v2 . You might expect forcing structured reasoning to improve accuracy. It did not. In my tests, config prompt v2 added 711 tokens and increased total cost, but it actually performed worse in Run 1 or identical in Run 2 to the baseline. Why? The model spent extra tokens "thinking out loud" summarizing the task description and detailing diagnostic steps but this produced no extra problem-solving value over the concise baseline prompt. Run this experiment on your own agent; the result may surprise you. Takeaway: In this experiment, on these 10 bug-fixing tasks, the more verbose diagnostic prompt increased token usage without improving task performance. Whether that generalizes to other models, other task types, or genuinely harder bugs is an open question, but it's a cheap thing to test before assuming "more structured reasoning" is automatically better. Finding: The Blind Execution Paradox What happens if you take away an agent's ability to test its own code? I ran config no run tests , an environment where the agent only had read file and edit file tools—no run tests tool. Remarkably, the agent's pass rate improved to 90% up from 70% , achieving a perfect 1.00 completion score. Even though the agent couldn't run tests itself, when my evaluator ran the strict pytest suite at the end, 9 out of 10 fixes were perfectly correct. A small caveat: the jump from 70% to 90% is really just two extra tasks passing. With only two runs, I can't be sure this gap is real and not just randomness. What I'm more confident about is the quality rating, which held at a perfect 1.00 in both runs. Without the ability to lean on a "guess-and-check" loop, the agent carefully read the source code and the test suite in its initial turn, identified the exact root cause, and made a single, precise edit. The trade-off? Blind execution used slightly fewer total tokens but took 1.0 second longer per task due to heavier reasoning per turn. Takeaway: In this experiment, removing the run tests tool didn't hurt performance; it may have helped, by forcing more careful upfront reasoning instead of a guess-and-check loop. My hypothesis, untested here since every task was single-file and localized, is that this trade-off flips for complex, multi-file bugs, where a testing loop becomes essential. That's a good next experiment for this framework, not a strong conclusion this dataset can support yet. Methodology: Sequence & Flow Analysis A correct answer can hide a terrible process. Did the agent get stuck in a dead loop? Did it wildly edit files without reading them first? My trajectory.py and tool flow.py evaluators analyze the shape of the execution trace. They look for missing verification steps or looped failures. If an agent manages to fix a bug but took 5 redundant edits to get there, a trajectory evaluator will flag it, whereas a simple outcome evaluator would happily mark it as a "pass." Finding: The Cost of Strict Efficiency In production, wrong answers are loud—users complain, tests fail, judges flag them. But expensive, slow, correct answers are quiet. They just slowly drain your API credits. My efficiency.py evaluator tracks latency, token consumption, USD cost, and turn counts per task. During my benchmark, I noticed persistent failures on two simple tasks task 002 and task 007 . Why? Because to force efficiency, I set a harsh harness constraint: max turns: 3 . The minimum viable trajectory read → edit → verify is exactly 3 turns. Any slight imprecision on the first try meant the agent hit the turn limit and failed, even if it had the correct logic. Takeaway: Evaluation harnesses must carefully distinguish between agent capability failures the model isn't smart enough and harness constraints the turn budget was just too tight . Methodology: CI Gates and Statistical Variance Once an agent can solve a task, that task graduates to the regression suite. regression.py allows me to compare two runs of the same config over time to ensure an update to the model or prompt didn't break previously solved tasks. As I found in my benchmarks, LLMs are naturally unpredictable. A single run is great for a quick CI smoke test, but for true regression testing, you need to run tasks multiple times so you don't mistake LLM randomness for a broken prompt. Methodology: Diff-based Quality Evaluation Finally, did the agent fix the actual bug, or did it write a lazy workaround? If a test fails, a lazy agent might simply delete the failing assert statement in the test file. To prevent this, fix quality.py acts as a specialized LLM judge. Crucially, I didn't just feed it the trace log, I captured the exact git diff of the agent's final state and passed that to the judge. The judge categorizes the result as a GENUINE fix addressing the root architectural bug or a WORKAROUND cheating the test suite . Across all my completed tests, my agents achieved a 100% Genuine Fix rate. Building a custom evaluation harness is a great learning exercise, but what tools exist in the ecosystem to help you do this at scale? If you don't want to build this from scratch, here is how the tooling landscape looks today: Tooling caveat:This landscape moves fast: ownership changes Promptfoo was acquired by OpenAI in March 2026, though it remains open-source and APIs get marked experimental NVIDIA's own docs flag NeMo's evaluation API as such . Treat this section as a snapshot, and check current docs before building on any of these. A common real-world pattern: Use Promptfoo or a custom script outcome.py as a cheap, rule-based CI gate at PR time. Use DeepEval to run scheduled, LLM-judged evaluations on sampled production traces. Finally, visualize those scores and trajectories in Langfuse or Confident AI to monitor for drift. Agent evaluation is an evolving discipline, not a solved checklist. By stepping away from simple single-turn accuracy and focusing on trajectories, tool correctness, and config-comparisons, you can build agents that don't just answer questions correctly—but act reliably in the real world. By structuring your evaluation around these 8 critical questions, and blending cheap rule-based CI gates with nuanced LLM-as-a-judge scoring, you can finally prove that your agent works not just in manual testing, but in production. If you want to dive deeper into the state of the art in agent evaluation, here are some excellent resources: